Today, after deploying metrics server, I checked the pod log and found a pile of errors:
The error information is as follows:
]# kubectl logs -f -n kube-system metrics-server-d8669575f-xl6mw I1202 09:09:31.217954 1 serving.go:312] Generated self-signed cert (apiserver.local.config/certificates/apiserver.crt, apiserver.local.config/certificates/apiserver.key) I1202 09:09:37.725863 1 secure_serving.go:116] Serving securely on [::]:443 E1202 09:09:49.807117 1 reststorage.go:135] unable to fetch node metrics for node "master": no metrics known for node E1202 09:09:49.807185 1 reststorage.go:135] unable to fetch node metrics for node "node1": no metrics known for node E1202 09:09:49.807202 1 reststorage.go:135] unable to fetch node metrics for node "node2": no metrics known for node E1202 09:09:50.940606 1 reststorage.go:160] unable to fetch pod metrics for pod linux40/nginx-deployment-7d8599fbc9-68pf8: no metrics known for pod E1202 09:09:53.825493 1 reststorage.go:135] unable to fetch node metrics for node "node1": no metrics known for node E1202 09:09:53.825540 1 reststorage.go:135] unable to fetch node metrics for node "node2": no metrics known for node E1202 09:09:53.825551 1 reststorage.go:135] unable to fetch node metrics for node "master": no metrics known for node E1202 09:10:05.976306 1 reststorage.go:160] unable to fetch pod metrics for pod linux40/nginx-deployment-7d8599fbc9-68pf8: no metrics known for pod E1202 09:10:21.291923 1 reststorage.go:160] unable to fetch pod metrics for pod linux40/nginx-deployment-7d8599fbc9-68pf8: no metrics known for pod E1202 09:10:31.601208 1 reststorage.go:135] unable to fetch node metrics for node "master": no metrics known for node E1202 09:10:31.601330 1 reststorage.go:135] unable to fetch node metrics for node "node1": no metrics known for node E1202 09:10:31.601353 1 reststorage.go:135] unable to fetch node metrics for node "node2": no metrics known for node E1202 09:10:31.610963 1 reststorage.go:160] unable to fetch pod metrics for pod kube-system/kube-flannel-ds-64qdh: no metrics known for pod E1202 09:10:31.611032 1 reststorage.go:160] unable to fetch pod metrics for pod linux40/magedu-tomcat-app1-deployment-6cd664c5bd-wprjb: no metrics known for pod
No valid error message was found when viewing the details of pod
]# kubectl describe pod metrics-server-6c97c89fd5-j2rql -n kube-system
Name: metrics-server-6c97c89fd5-j2rql
Namespace: kube-system
Priority: 2000000000
Priority Class Name: system-cluster-critical
Node: node2/192.168.64.112
Start Time: Thu, 02 Dec 2021 16:50:50 +0800
Labels: k8s-app=metrics-server
pod-template-hash=6c97c89fd5
Annotations: <none>
Status: Running
IP: 10.244.2.61
IPs:
IP: 10.244.2.61
Controlled By: ReplicaSet/metrics-server-6c97c89fd5
Containers:
metrics-server:
Container ID: docker://eac4a2db02ca75315047eb778b7d3e1d7543d10ed6d33b4b1eddb006f824e34e
Image: mirrorgooglecontainers/metrics-server-amd64:v0.3.6
Image ID: docker://sha256:9dd718864ce61b4c0805eaf75f87b95302960e65d4857cb8b6591864394be55b
Port: 4443/TCP
Host Port: 0/TCP
Args:
--cert-dir=/tmp
--secure-port=4443
--kubelet-preferred-address-types=InternalIP
--kubelet-use-node-status-port
--kubelet-insecure-tls
State: Waiting
Reason: CrashLoopBackOff
Last State: Terminated
Reason: Error
Exit Code: 2
Started: Thu, 02 Dec 2021 16:51:10 +0800
Finished: Thu, 02 Dec 2021 16:51:11 +0800
Ready: False
Restart Count: 2
Liveness: http-get https://:https/livez delay=0s timeout=1s period=10s #success=1 #failure=3
Readiness: http-get https://:https/readyz delay=0s timeout=1s period=10s #success=1 #failure=3
Environment: <none>
Mounts:
/tmp from tmp-dir (rw)
/var/run/secrets/kubernetes.io/serviceaccount from metrics-server-token-4xrbc (ro)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
tmp-dir:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium:
SizeLimit: <unset>
metrics-server-token-4xrbc:
Type: Secret (a volume populated by a Secret)
SecretName: metrics-server-token-4xrbc
Optional: false
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s
node.kubernetes.io/unreachable:NoExecute for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled <unknown> default-scheduler Successfully assigned kube-system/metrics-server-6c97c89fd5-j2rql to node2
Normal Pulled 16s (x3 over 34s) kubelet, node2 Container image "mirrorgooglecontainers/metrics-server-amd64:v0.3.6" already present on machine
Normal Created 16s (x3 over 34s) kubelet, node2 Created container metrics-server
Normal Started 16s (x3 over 34s) kubelet, node2 Started container metrics-server
Warning BackOff 8s (x5 over 32s) kubelet, node2 Back-off restarting failed container
Because the CA certificate does not sign the IP of each node when deploying the cluster, when the metrics server requests through IP, it will prompt that the signed certificate does not have a corresponding IP (error: x509: cannot validate certificate for 192.168.33.11 because it doesn’t contain any IP SANS),
We can add a — kubelet secure TLS parameter to skip certificate verification:
apiVersion: apps/v1
kind: Deployment
metadata:
name: metrics-server
namespace: kube-system
labels:
k8s-app: metrics-server
spec:
selector:
matchLabels:
k8s-app: metrics-server
template:
metadata:
name: metrics-server
labels:
k8s-app: metrics-server
spec:
serviceAccountName: metrics-server
volumes:
- name: tmp-dir
emptyDir: {}
containers:
- name: metrics-server
image: mirrorgooglecontainers/metrics-server-amd64:v0.3.6
imagePullPolicy: IfNotPresent
command:
- /metrics-server
- --kubelet-insecure-tls //skip tls
- --kubelet-preferred-address-types=InternalIP //Using internal IP communication
volumeMounts:
- name: tmp-dir
mountPath: /tmp
resources:
limits:
cpu: 300m
memory: 200Mi
requests:
cpu: 200m
memory: 100Mi