Updating Kubernetes Certificates: Easy Peasy!(En)

Recently, one of my client asked about updating Cluster certificates upon expiration, and I thought it would be a good opportunity to share our previous update process with everyone.

This article will focus on scenarios where ca.crt/etcd-ca certificates have not expired yet, making the process relatively straightforward. Just follow the steps outlined in this article, and you’ll smoothly extend the validity period.

It’s time to renew the certificate!

The article will cover the following topics:

1. Overview and Precautions
2. Certificate Inspection and Backup
3. Certificate Renewal
4. Conclusion

1. Overview and Precautions

When using kubeadm to build a K8S Cluster, certificates are automatically generated for all components, with a default expiration of 1 year. Upon expiration, you may encounter connection issues to localhost:8080.

If regular K8S version upgrades are performed, the certificate will be updated as part of the upgrade process.

The default config paths are as follows:

• Working directory: /etc/kubernetes
• Certificate directory: /etc/kubernetes/pki

(1) Root CA

(2) Certificates from CA

(3) User account certificates

※ Considerations:

• If using an external cluster CA, it’s advisable to extend the expiration period(maybe 10 years?) because unlike the API server certificate, the cluster CA replacement requires multiple restarts of Pods using service accounts, leading to more involved operations.
• Before proceeding, ensure relevant backup are taken, including backing up configuration files or taking snapshots of VMs.

2. Certificate Inspection and Backup

#--------------------------------------
# S2-1. Check the expiration date of the current certificate
#--------------------------------------
[master]# kubeadm certs check-expiration
OR
[master]# openssl x509 -in /etc/kubernetes/pki/apiserver.crt -noout -text | grep ' Not '
[master]# for tls in `find /etc/kubernetes/pki -name "*.crt"`; do echo $tls; openssl x509 -in $tls -text | grep Not; done

#--------------------------------------
# S2-2. Backup entire config dir
#--------------------------------------
[master]# cp -rp /etc/kubernetes /root/old_k8s_config

3. Certificate Renewal

#--------------------------------------
# S3-1. Renew certificate (master)
#--------------------------------------
[master]# kubeadm certs renew all 
[master]# kubeadm certs check-expiration
[master]# ls -al /etc/kubernetes/pki/

#--------------------------------------
# S3-2. Re-generate all of configurations (master only)
#--------------------------------------
[master]# rm -rf /etc/kubernetes/*.conf
[master]# kubeadm init --kubernetes-version=v1.28.8 phase kubeconfig all
[master]# ls -al /etc/kubernetes/

4. Conclusion

Updating certificates is something you’ll do regularly. Make sure to back up any files you’re going to change. If you’re working with VMs, it’s smart to take snapshots so you can go back if something goes wrong. Sometimes, updating certificates might cause issues, but in my tests, simply restarting the node fixed them.

After you update certificates, if you see any old containers still running but not doing anything useful, just delete them with a command.

Overall, as long as you’re keeping up with upgrades, you shouldn’t stress too much about certificates expiring.