ZK: 3.4.9 (3 nodes)

kubernetes: 1.17.2

The container image is self encapsulated using the CentOS 7 + ZK installation package

Problem description
the three nodes of ZK are all started, and the cluster is also built to provide external services normally. However, the node status is abnormal when you query zkserver.sh status, as shown below

$./bin/zkserver.sh status
zookeeper JMX enabled by default
using config:/usr/zookeeper-3.4.9/bin /../conf/zoo. CFG
error contacting service. It is possible not running
you can see the ZK process using JPS

$JPS
26 quorumpeermain
120300 JPS
troubleshooting process
according to reference article 1, modify bin/zkserver.sh to view the detailed process of the status subcommand

STAT=`”$JAVA” “-Dzookeeper.log.dir=${ZOO_ LOG_ DIR}” “-Dzookeeper.root.logger=${ZOO_ LOG4J_ PROP}” \ -cp “$CLASSPATH” $JVMFLAGS org.apache.zookeeper.client.FourLetterWordMain \ $clientPortAddress $clientPort srvr 2>/Dev/null \ | $grep mode `
add 2 & gt/ Dev/null and the following pipeline filter statements are removed. Check the output of the srvr instruction and get the following

Exception in thread “main” java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.refl ect.Method.invoke(Method.java:498)
at sun.instrument.InstrumentationImpl.loadClassAndStartAgent(InstrumentationImpl.java:386)
at sun.instrument.InstrumentationImpl.loadClassAndCallPremain(InstrumentationImpl.java:401)
Caused by: java.net.BindException: Address already in use
at sun.nio.ch.Net.bind0(Native Method)
at sun.nio.ch.Net.bind(Net.java:433)
at sun.nio.ch.Net.bind(Net.java:425)
at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223)
at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
at sun.net.httpserver.ServerImpl.< init>( ServerImpl.java:100)
at sun.net.httpserver.HttpServerImpl.< init>( HttpServerImpl.java:50)
at sun.net.httpserver.DefaultHttpServerProvider.createHttpServer(DefaultHttpServerProvider.java:35)
at com.sun.net.httpserver.HttpServer.create(HttpServer.java:130)
at io.prometheus.jmx.shaded.io.prometheus.client.exporter.HTTPServer.< init>( Httpserver. Java: 179)
at Io. Prometheus. JMX. Shaded. Io. Prometheus. JMX. Javaagent. Premain (javaagent. Java: 31)… 6 more
fat error in native method: processing of – javaagent failed
as mentioned in reference article 1, it is address already in use, but TA did not say which port it is 🤔

And ZK is running normally. It’s not normal for 2181/3888 to be occupied

The following is the process of taking a chance. According to the problem mentioned in reference article 2, the datalogdir directory is not successfully created, or the myid path is incorrect… Can these configurations be started successfully?

Reference article 3 mentioned the firewall, um… There is no firewall in the container. In addition, there is no need for/etc/hosts. The address nodes in server. X can communicate with each other (of course, I tried, as expected) 🤔).

Solution
the final solution is to refer to Article 5. In order to monitor ZK containers in a container environment, we use JMX_ Exporter, so the jvmflags variable in zkserver.sh is modified

JVMFLAGS=”$JVMFLAGS -javaagent:$JMX_ DIR/jmx_ prometheus_ javaagent-0.15.0.jar=19105:$JMX_ Dir/zookeeper. Yaml “
comment out this line and execute zkserver.sh status

In addition, in addition to this reason, another reason is that our zoo.cfg configuration is not standard, as shown below

clientPort:2181
server.0:zk-ha-test-busi-kafka-0.zk-ha-test-busi-kafka-svc.zjjpt-zk.svc.cs-hua.hpc:2888:3888
server.1:zk-ha-test-busi-kafka-1.zk-ha-test-busi-kafka-svc.zjjpt-zk.svc.cs-hua.hpc:2888:3888
server.2:zk-ha-test-busi-kafka-2.zk-ha-test-busi-kafka-svc.zjjpt-zk.svc.cs-hua.hpc:2888:3888
dataLogDir:/data/zk-ha-test-busi-kafka-0/l Og
dataDir:/data/zk-ha-test-busi-kafka-0
##… Omit
use colon: instead of equal sign =, ZK can run normally 😱.

This causes problems in parsing the clientport variable from zoo.cfg when executing the status subcommand in zkserver.sh

Clientport = ` $grep “^ [[: Space:]] * clientport [^ [: Alpha:]]” $zoocfg “| sed – E’s /. * =// ‘`
the following sed instruction is segmented by =. The result is clientport: 2181, which is an illegal value

Modify the format of zoo.cfg