배경
관리하던 학교 Kubernetes(이하 K8s) 클러스터에 접속해서 관리를 하려고 했더니 다음처럼 나왔다.
$ kubectl get pods
Unable to connect to the server: x509: certificate has expired or is not yet valid: current time 2023-07-14T01:22:53Z is after 2023-07-10T11:45:22Z그래서 이를 해결하고자 살짝의 삽질을 했다 (한 10분 정도니까 삽질이라고 하기도 좀 그렇다...)
인증서 확인
먼저 정말로 X509 인증서가 만료되었는지 확인하자.
sudo kubeadm certs check-expiration
[check-expiration] Reading configuration from the cluster...
[check-expiration] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
[check-expiration] Error reading configuration from the Cluster. Falling back to default configuration
CERTIFICATE EXPIRES RESIDUAL TIME CERTIFICATE AUTHORITY EXTERNALLY MANAGED
admin.conf Jul 10, 2023 11:45 UTC <invalid> ca no
apiserver Jul 10, 2023 11:45 UTC <invalid> ca no
apiserver-etcd-client Jul 10, 2023 11:45 UTC <invalid> etcd-ca no
apiserver-kubelet-client Jul 10, 2023 11:45 UTC <invalid> ca no
controller-manager.conf Jul 10, 2023 11:45 UTC <invalid> ca no
etcd-healthcheck-client Jul 10, 2023 11:45 UTC <invalid> etcd-ca no
etcd-peer Jul 10, 2023 11:45 UTC <invalid> etcd-ca no
etcd-server Jul 10, 2023 11:45 UTC <invalid> etcd-ca no
front-proxy-client Jul 10, 2023 11:45 UTC <invalid> front-proxy-ca no
scheduler.conf Jul 10, 2023 11:45 UTC <invalid> ca no
CERTIFICATE AUTHORITY EXPIRES RESIDUAL TIME EXTERNALLY MANAGED
ca Jul 07, 2032 11:45 UTC 8y no
etcd-ca Jul 07, 2032 11:45 UTC 8y no
front-proxy-ca Jul 07, 2032 11:45 UTC 8y no정말로 만기되었다. 7월 10일자로 이런 저런 클러스터의 인증서가 만료되었다. 신기한게, ca와 etcd-ca 처럼 ca의 인증서는 10년짜리인 것 같다. 아무튼... 만료되었다.
인증서 연장
인증서를 연장하면 된다. https://stackoverflow.com/questions/49885636/kubernetes-expired-certificate
Kubernetes: expired certificate
Our Kubernetes 1.6 cluster had certificates generated when the cluster was built on April 13th, 2017. On December 13th, 2017, our cluster was upgraded to version 1.8, and new certificates were gen...
stackoverflow.com
를 확인하면 상당히 복잡해보이지만, 공식 문서 https://kubernetes.io/docs/tasks/administer-cluster/kubeadm/kubeadm-certs/
Certificate Management with kubeadm
FEATURE STATE: Kubernetes v1.15 [stable] Client certificates generated by kubeadm expire after 1 year. This page explains how to manage certificate renewals with kubeadm. It also covers other tasks related to kubeadm certificate management. Before you begi
kubernetes.io
는 훨씬 간단하게 소개를 한다.
$ sudo kubeadm certs renew all
[renew] Reading configuration from the cluster...
[renew] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
[renew] Error reading configuration from the Cluster. Falling back to default configuration
certificate embedded in the kubeconfig file for the admin to use and for kubeadm itself renewed
certificate for serving the Kubernetes API renewed
certificate the apiserver uses to access etcd renewed
certificate for the API server to connect to kubelet renewed
certificate embedded in the kubeconfig file for the controller manager to use renewed
certificate for liveness probes to healthcheck etcd renewed
certificate for etcd nodes to communicate with each other renewed
certificate for serving etcd renewed
certificate for the front proxy client renewed
certificate embedded in the kubeconfig file for the scheduler manager to use renewed
Done renewing certificates. You must restart the kube-apiserver, kube-controller-manager, kube-scheduler and etcd, so that they can use the new certificates.간단하게 renewal이 되었다. 이제 또 다시 인증서를 확인하면
$ sudo kubeadm certs check-expiration
[check-expiration] Reading configuration from the cluster...
[check-expiration] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
W0714 01:51:36.018912 1915378 configset.go:177] error unmarshaling configuration schema.GroupVersionKind{Group:"kubeproxy.config.k8s.io", Version:"v1alpha1", Kind:"KubeProxyConfiguration"}: strict decoding error: unknown field "udpIdleTimeout"
[check-expiration] Error reading configuration from the Cluster. Falling back to default configuration
CERTIFICATE EXPIRES RESIDUAL TIME CERTIFICATE AUTHORITY EXTERNALLY MANAGED
admin.conf Jul 13, 2024 01:51 UTC 364d ca no
apiserver Jul 13, 2024 01:51 UTC 364d ca no
apiserver-etcd-client Jul 13, 2024 01:51 UTC 364d etcd-ca no
apiserver-kubelet-client Jul 13, 2024 01:51 UTC 364d ca no
controller-manager.conf Jul 13, 2024 01:51 UTC 364d ca no
etcd-healthcheck-client Jul 13, 2024 01:51 UTC 364d etcd-ca no
etcd-peer Jul 13, 2024 01:51 UTC 364d etcd-ca no
etcd-server Jul 13, 2024 01:51 UTC 364d etcd-ca no
front-proxy-client Jul 13, 2024 01:51 UTC 364d front-proxy-ca no
scheduler.conf Jul 13, 2024 01:51 UTC 364d ca no
CERTIFICATE AUTHORITY EXPIRES RESIDUAL TIME EXTERNALLY MANAGED
ca Jul 07, 2032 11:45 UTC 8y no
etcd-ca Jul 07, 2032 11:45 UTC 8y no
front-proxy-ca Jul 07, 2032 11:45 UTC 8y no넉넉하게 1년짜리 인증서를 만들어줬다.
로그인 (마스터 노드)
하지만 이러고는 kubectl 명령어를 입력하면 안될것이다.
$ kubectl get pods
error: You must be logged in to the server (Unauthorized)왜냐? 우리가 아까 certificates 중 admin.conf를 renewal 했는데, 이거랑 지금 내가 내 ~/.kube/config에 있는 인증서랑 다르기 때문이다. 따라서 이걸 다시 반영해줘야한다.
$ sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
cp: overwrite '/home/isu/.kube/config'? y
$ sudo chown $(id -u):$(id -g) $HOME/.kube/config이렇게 해주면 기존의 config를 새로 덮어쓰고 사용할 수 있게 된다. 이제 kubectl 이 잘 동작할 것이다.
워커 노드 설정
하지만 모두 예상했던 것 처럼, 우리는 처음에 Kubernetes를 사용할 때 master 노드에서 생성된 admin.conf를 워커 노드로도 보낸다. 물론 옵션이지만, 워커 노드에서 kubectl을 하고 싶으면 그렇게 해야한다.
지금 이렇게 되면 마스터 노드에서는 인증서가 갱신된 config를 가지지만, 워커노드는 그렇지 않다. 따라서 마스터 노드의 config 파일을 보내줘야한다. 간단히 scp로 보내자
scp /home/isu/.kube/config isu@172.20.41.152:/home/isu/.kube/디렉토리랑 워커 노드를 잘 설정해서 보내면 된다.
