Setup Multi Node Kubernetes Cluster

MAYANK VARSHNEY

6 min readJun 26, 2020

Working on a containerized storage for containers which is orchestrated by Kubernetes, most of our tasks, be it development, testing and demo require us to setup and modify the nodes in Kubernetes cluster. In addition, a multi-node cluster is a must as we go beyond the initial development and testing, to explore the high availability, scale, performance and upgrade aspects.
While minikube and minishift provide an easy way to setup kubernetes single node cluster — for multi-node cluster the fastest ways to get going are usually cloud or hosted solutions. kubeadm is the closest we can get to easily setup a cluster.
For creating a cluster we need to run multiple OS/Nodes. Here, we have four ways , we can run the OS on the top of Laptop, Virtual Machine, AWS EC2 Service, and Server.
Here I am using a Oracle VirtuaBox and on top of it I’ll run the RHEL8 Linux OS, one master node and two slaves to create a entire cluster(nodes varies upon the uses cases and can be added accordingly). We'll start with installing the redhat8 cli on top of virtualbox.

Create a new virtual machine in VM.

2. Provide atleast 2gb ram to the OS.

3. Click on next and follow up the default installation settings till create.

4. Settings >Storage > Select the iso file from the location.

you should have the iso file downloaded on your system of rhel8.

5. Select start and begin the installlation (Select ‘minimal install’ in software selection).

6. After installation complete. Do the following changes :

put the hard-disk priority on top.

Select minimum two CPU’s.

Now we are good to go.

NOTE: We need some programs to run on the OS, like “Container-Engine”, here we are going to use “docker-ce”, kubelet, and some other prerequisites . These programs are required in master node as well as slave node, so to save our time we will setup all the programs in one OS and then clone the others with the similar settings.

Now mount the iso file or dvd to one folder.

Setup yum to install required softwares.

Install net-tools, vim, and httpd for future preferences.

yum install net-tools vim httpd

In my case software's are already installed.

Install Docker in your RHEL8 Version.

yum install docker-ce — nobest (put two hyphen before nobest).

Note:

Disable the firewall security in redhat OS to allow the incoming ports.

systemctl stop firewalld
systemctl disable firewalld

Make the dvd mount permanent so that after reboot no need to mount again and again.

vi /etc/rc.d/rc.local

type “chmod +x /etc/rc.d/rc.local” after making the changes in file.

Install kubectl, kubeadm, kubelet using yum.

Type the following command to install the above softwares.

For working of these software’s properly SELinux Security must be stopped or disabled.

Type “vi /etc/selinux/config” and make the above changes.

Now start the docker services to start the container engine.

systemctl start docker
systemctl enable docker

Note: We need to change the CGroup Driver to “systemd” in redhat 8 to manage everything properly.

follow the above steps to apply the “systemd” driver .

We can check now if docker is working properly by using “docker info” command and also check the cgroup driver is changed or not.

We also need to disable the swap services to run the Kubernetes perfectly on Linux OS.

Type “vi /etc/fstab” and comment the last line to disable the swap services to run kubelet .

Install the “iproute-tc” software on linux that manages the traffic control.

Linux Nodes iptables to see brigged traffic correctly its configuration must be set to 1. Follow the below snap :

“sysctl — system” to set the configuration.

Start and enable the kubelet services.

systemctl start kubelet
systemctl enable kubelet

services will start when master node will connect..

Our setup part is now almost complete, for creating multiple slaves and master nodes we'll clone the above OS and create the new nodes.

simply clone the OS . In my case I Cloned one master node and two slave nodes, shown above.

Set the host name of each node so that in future we can connect or ping with the hostnames (as ip is not always same) .

To ping the nodes with the hostname the details must be updates in the “/etc/hosts” file or in other words DNS server must be created.

Type “vi /etc/hosts” and save the changes. Do the same updation in other nodes as well and then try pinging to each node to check the connectivity.

Now initialize the “Kubeadm” to the master node to start the Kubernetes in Linux Server.

— pod-network-cidr will set the desired range of the pods.

After Kubeadm init properly, to connect the services we need to apply the following commands.

after completion, we can check the kubectl services.

To join the worker nodes we need to run the following in each node as root.

Copy the same command and apply on other nodes as well to connect .

We need to provide one extra network layer to connect between the pods to create a multi node cluster.

Here I use “Flannel” Overlay Network Provider.

Apply the command with the link to provide the overlay network.

After all the steps done successfully , we can see that our multi node cluster is created successfully.

Copy the config file (contains the certifications and other files required to connect to the cluster from client source) to the windows desktop in any folder.

In my case i use windows as my client source and there i have minikube and kubectl services installed on top of virtualbox , now i will connect to my multi node cluster using the config file on windows cmd.

cluster connects to client successfully.

One Short Example to show the use-case of Multi-Node Kubernetes Cluster:

Launch a ReplicaSet with 3 replicas using httpd image .

3 pods will be created in different slaves randomly and the ip will be provided within the range that we mentioned before.

Here we can check that two pods are created in slave2 and one pod in slave1 and we launch the replica-set from master node . So this is one short example of multi-node Kubernetes cluster.

We can verify where the pods are running by manually checking the nodes.

We can verify from above snap the management of multi node cluster.

Also we can Expose the Kubernetes services to the client source.

It provides a random port number through which we can access any of the pods service by using the node/s IP address followed by the port number provided.