Essential Guide for K8S Newbies — Part 2: Setting Up Your First K8S Cluster(En)

7 min readDec 8, 2023

In the previous article, we completed the deployment of HAproxy and Keepalived. In this article, we will delve into the actual process of building the Kubernetes Cluster.

This article aims to guide you through the process of continuing the construction of your first Kubernetes Cluster, building upon the content shared in the previous post. If you haven’t read the previous one, you can click on the following link:

https://weng-albert.medium.com/essential-guide-for-k8s-newbies-part-1-getting-your-rhel-environment-ready-en-1a72e8f122fc

For those of you currently reading this article, you can seamlessly follow along to construct your container platform step by step. Without further ado, here are the key sections we’ll be covering in this chapter:

Pre-install Config for Nodes
Software Package Installation
Control-Plane Deployment
Adding Compute Nodes
Conclusion

1. Pre-install Config for Nodes

The following content should be configured on each node based on its role

#------------------------------------------------------
# S1-1. FW rules (control-planes)
#------------------------------------------------------
[root]# systemctl enable --now firewalld ; systemctl status firewalld
[root]# vim set_firewall_master.sh
#!/bin/bash
firewall-cmd --zone=public --add-service=kube-apiserver --permanent
firewall-cmd --zone=public --add-service=etcd-client --permanent
firewall-cmd --zone=public --add-service=etcd-server --permanent
firewall-cmd --zone=public --add-port=10250/tcp --permanent
firewall-cmd --zone=public --add-port=10251/tcp --permanent
firewall-cmd --zone=public --add-port=10252/tcp --permanent
firewall-cmd --zone=public --add-port=179/tcp --permanent
firewall-cmd --reload

[root]# chmod +x set_firewall_master.sh 
[root]# ./set_firewall_master.sh; firewall-cmd --list-all

#------------------------------------------------------
# S1-2. FW rules (workers)
#------------------------------------------------------
[root]# systemctl enable --now firewalld ; systemctl status firewalld
[root]# vim set_firewall_worker.sh
#!/bin/bash
firewall-cmd --zone=public --add-port=10250/tcp --permanent
firewall-cmd --zone=public --add-port=30000-32767/tcp --permanent
firewall-cmd --zone=public --add-port=179/tcp --permanent
firewall-cmd --reload

[root]# chmod +x set_firewall_worker.sh ; 
[root]# ./set_firewall_worker.sh; firewall-cmd --list-all

#------------------------------------------------------
# S1-3. time sync (all nodes)
#------------------------------------------------------
[root]# systemctl enable --now chronyd ; chronyc sources

#------------------------------------------------------
# S1-4. Turn off SWAP (all nodes)
#------------------------------------------------------
[master]# swapoff -a && sed -i '/ swap / s/^\(.*\)$/#\1/g' /etc/fstab

#------------------------------------------------
# S1-5. Lode module (all nodes)
# By default, Linux prohibits data forwarding, 
# and to address this, you need to load br_netfilter.
#------------------------------------------------
[root]# vim /etc/modules-load.d/k8s.conf
overlay
br_netfilter

[root]# modprobe overlay ; modprobe br_netfilter

[root]# vim /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-iptables  = 1
net.ipv4.ip_forward                 = 1
net.bridge.bridge-nf-call-ip6tables = 1

[root]# sysctl --system

2. Software Package Installation

In this section, we will install the following software packages:

CRIO
kubelet
kubeadm
kubectl

#------------------------------------------------
# S2-1. Install CRIO (all k8s nodes)
#------------------------------------------------
[root]# export VERSION=1.27
[root]# curl -L -o /etc/yum.repos.d/devel:kubic:libcontainers:stable.repo https://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable/CentOS_8/devel:kubic:libcontainers:stable.repo
[root]# curl -L -o /etc/yum.repos.d/devel:kubic:libcontainers:stable:cri-o:$VERSION.repo https://download.opensuse.org/repositories/devel:kubic:libcontainers:stable:cri-o:$VERSION/CentOS_8/devel:kubic:libcontainers:stable:cri-o:$VERSION.repo

[root]# yum install cri-o -y
[root]# systemctl enable --now crio ; systemctl status crio
[root]# crio version
[root]# yum list cri-o --showduplicates|sort -r > crio.version

#------------------------------------------------
# S2-2. Modify the cgroup for CRIO
# When using systemd on Linux, it creates a cgroup. 
# It is essential to ensure that CRIO, kubelet, and 
# systemd all use the same cgroup to avoid unexpected issues. 
# Configure both CRIO and kubelet to use systemd as the cgroup 
# driver for a more stable system
#------------------------------------------------
[root]# cat /etc/crio/crio.conf | grep cgroup_manager
# cgroup_manager = "systemd"
=> comment out "#"

[root]# systemctl restart crio ; systemctl status crio

#------------------------------------------------
# S2-3. kubelet,kubeadm,kubectl (all k8s nodes)
#------------------------------------------------
[root]# vim /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://packages.cloud.google.com/yum/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=0
repo_gpgcheck=0
gpgkey=https://packages.cloud.google.com/yum/doc/yum-key.gpg https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg

[root]# yum clean all ; yum repolist

#----------------------------
# S2-4. Check available version
# We choose 1.27.x
#----------------------------
[root]# yum list kubelet --showduplicates|sort -r > kubelet.version
[root]# yum list kubeadm --showduplicates|sort -r > kubeadm.version
[root]# yum list kubectl --showduplicates|sort -r > kubectl.version

kubeadm: 1.27.6-0
kubectl: 1.27.6-0
kubelet: 1.27.6-0

#------------------------------------------------
# S2-5. Install kubeadm, kubelet, kubectl (all nodes)
#------------------------------------------------
[root]# yum install kubelet-1.27.6-0 kubeadm-1.27.6-0 kubectl-1.27.6-0
[root]# systemctl enable --now kubelet ; systemctl status kubelet

3. Control-Plane Deployment

#------------------------------------------------
# S3-1. Create first control plane node (master01)
# PS. if any error happen, use "kubeadm reset" 
#------------------------------------------------
[root@master01]# kubeadm init --control-plane-endpoint "10.107.88.9:6443" --upload-certs --pod-network-cidr=192.168.0.0/16
※ --upload-certs: Automatically store the certificates used by the control plane in a secret named "kubeadm-certs." If you want to use your own certificates, remove this parameter.

** RESULT **
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

Alternatively, if you are the root user, you can run:

  export KUBECONFIG=/etc/kubernetes/admin.conf

kubeadm join 10.107.88.9:6443 --token 1s990r.atf22thmail583k6 \
        --discovery-token-ca-cert-hash sha256:68cb14ed46cc2454d11cc9127856ee2df3cf45cb9d7a7c34fd97c624e86ce6ab \
        --control-plane --certificate-key 0e5614062d18ae699e4f79609992908cd8e25b9409abdeeee0bb842dfcb28208
        
kubeadm join 10.107.88.9:6443 --token 1s990r.atf22thmail583k6 \
        --discovery-token-ca-cert-hash sha256:68cb14ed46cc2454d11cc9127856ee2df3cf45cb9d7a7c34fd97c624e86ce6ab
        
※ If you've forgotten the join command, use the following command to display it again：
[master]# kubeadm token create --print-join-command

#------------------------------------------------
# S3-2. Verify (master01)
#------------------------------------------------
[master]# kubectl get nodes
[master]# kubectl get pod -n kube-system (Check master components)
[master]# crictl images
[master]# crictl ps

[master]# nc -v 10.107.88.9 6443 (Test API server LB)
Ncat: Version 7.70 ( https://nmap.org/ncat )
Ncat: Connected to 10.107.88.9:6443.

#------------------------------------------------
# S3-3. The token has a 24-hour expiration period. 
# If you want to add new nodes after 24 hours, you 
# need to regenerate the token on master01. After 
# regenerating it, you must perform a cluster reset
# and then reinitialize it
# token expire: 24HR
# certificate-key: 2HR
#------------------------------------------------
※ After more than 24 hours, 
the message that appears after executing 'join' on master02 
(as shown in the image)
Solution:
(1) Rebuild the cluster: Execute kubeadm reset on master01 to start fresh and rebuild the entire cluster.
(2) Create a non-expiring token before initialization:`kubeadm token create --ttl 0 --print-join-command`

#------------------------------------------------
# S3-4. Create a non-expiring token
#------------------------------------------------
[master]# kubeadm token create --ttl 0 --print-join-command (worker)
kubeadm join 10.107.88.9:6443 --token 19j12t.veocwjglauw623fb --discovery-token-ca-cert-hash sha256:68cb14ed46cc2454d11cc9127856ee2df3cf45cb9d7a7c34fd97c624e86ce6ab

[master]# kubeadm init phase upload-certs --upload-certs --v=999
6f4a4de6273d5dbd4288814e4156b98334ca927cfdcca51beec7f37a7a9463bd

#------------------------------------------------
# S3-5. Combining the worker command to generate the master join command:
#  (master02, maser03)
#------------------------------------------------
[master]# kubeadm join 10.107.88.9:6443 --token 19j12t.veocwjglauw623fb --discovery-token-ca-cert-hash sha256:68cb14ed46cc2454d11cc9127856ee2df3cf45cb9d7a7c34fd97c624e86ce6ab --certificate-key 6f4a4de6273d5dbd4288814e4156b98334ca927cfdcca51beec7f37a7a9463bd --control-plane --v=999

#------------------------------------------------
# S3-6. Verify token (master01)
#------------------------------------------------
[master01]# kubeadm token list

#------------------------------------------------
# S3-7. Control-plane is done
#------------------------------------------------
[master]# kubectl get nodes

The control-plane setup is now complete. In practical scenarios, after obtaining the results from S3–1, you can directly execute the following steps on other nodes. The subsequent steps are necessary because I resumed deploying additional nodes after more than 24 hours since setting up the first node. In practice, it’s common to deploy all three control-plane nodes in one go. If time is a constraint, using the reset command allows for a quick start from scratch.

4. Adding Compute Nodes

Adding a compute node is also straightforward. Simply copy the output from S3–1 and execute it on the compute node.

#------------------------------------------------
# S4-1. add workers (all workers)
#------------------------------------------------
[worker]# kubeadm join 10.107.88.9:6443 --token p9gw5v.lepwj4nbkrshrqpl

5. Conclusion

This article quickly guides everyone through the process of setting up a Kubernetes Cluster, demonstrating that the overall procedure is not too challenging. However, from my own experience, the most challenging part of this solution begins after the completion of the setup. In practice, the current state is insufficient for actual service use.

The next article will explain several foundational services that I typically add right after the installation. These services enable the freshly established cluster to further develop operational capabilities or undergo management.

Thank you all for reading this article, and I’ll see you in the next one!