K3S by using Docker Compose

5 min readJul 1, 2023

Last week we received a job from a customer who wants to deploy their own web application on the cloud but the customer doesn’t have the budget to build a large Kubernetes cluster. The good news is the web application does not need to support the large load, so just a single VM is enough.

We began to study various information on the internet. It was concluded that K3S would be used for this task. Yes, we tried it and we are very satisfied. Even now we still use it in our production environment. That’s why we wrote and share this article.

Why we use K3S instead of Docker Compose?

There are several reasons why we decided to use K3S instead of deploying applications directly to Docker Compose:

We want to use GitOps to deploy applications. Yes, ArgoCD can be installed on K3S and this will make our lifes easier to deploy the applications into K3S.
If in the future the system needs to scale bigger, we will make the transition from K3S to Kubernetes easier because it uses GitOps.
Need to use an existing tech stack on Kubernetes such as Loki for viewing logs, Prometheus for monitoring, Cert Manager for Let’s Encrypt certificates.
K3S itself is a docker image which makes installation even easier. We also use Docker Compose to help us install the K3S, this is the major reason why we choose K3S instead of using other distributions.

Source Code

We develop the demo source code and keep it on github as follows.

K3S-Demo — It is a Docker Compose source code for the K3S deployment. ArgoCD will be initially installed along with the K3S.
K3S-Demo-Addons — It is the repository that ArgoCD keeps syncing the Kubernetes manifests and deploy into K3S cluster.

Provisioning the VM to deploy K3S

We use Google Compute Engine (GCE) to install the K3S.

e2-standard-4, 4 vCPU 16GB RAM, 100GB disk
Ubuntu OS, ubuntu-2004-lts
1 public IP

Install Docker using command shown below.

curl https://releases.rancher.com/install-docker/20.10.sh | sudo sh

Install Docker Compose using command shown below.

sudo apt install docker-compose

Install kubectl using command shown below.

sudo snap install kubectl --classic

K3S deployment

Now it is the time to deploy the K3S into the VM that we provisioned earlier.

Clone the source code from the K3S-Demo git repository. Once the source code is cloned then please change directory to the K3S-Demo

git clone https://github.com/its-knowledge-sharing/K3S-Demo.git
cd K3S-Demo

We need to create the .env file for initializing the required environment variables prior to start the Docker Compose. Actually, we can copy .env from from .env-template. Please note that the values in the .env-template are only for the demonstration. We must not keep any secrets in the Git repositories. Please use the command below to create the .env file.

cp .env-template .env

Let’s start the Docker Compose by running the start.bash script as shown in the command below.

./start.bash

Once Docker Compose is started, we will see the kubeconfig.yaml file in the current directory. Later we will be using it along with the kubectl command in order to communicate to K3S.

We will need to export the KUBECONFIG environment variable “export KUBECONFIG=kubeconfig.yaml” before using the kubectl command.

Wait for no longer than 10 minutes, we will see the applications in the K3S-Demo-Addons are synced into the K3S by ArgoCD as shown below.

K3S persistent volume

Please take a look volumes section in this file docker-compose.yaml to understand how we persist the data, no data loss even we restart the K3S containers or VM.

server:
...
...    
  volumes:
      - .:/output
      - ./addons/initial-secrets.yaml:/var/lib/rancher/k3s/server/manifests/initial-secrets.yaml
      - ./addons/addons-argocd.yaml:/var/lib/rancher/k3s/server/manifests/addons-argocd.yaml
      - k3s-server:/var/lib/rancher/k3s
...
...

agent:
...
...
  volumes:
     - k3s-agent:/etc/rancher/node
      - ${STORAGE_DIR}:/var/lib/rancher/k3s/storage

Exposing the NGginx Ingress Controller ports

Please take a look ports section in this file docker-compose.yaml to understand how we map the VM ports to NGinx Ingress Controller NodePort. Don’t forget to allow internet accessible for both port 443 and 80. Port 6443 should be configured to only accessible from localhost.

ports:
    - 6443:6443 # Should change this to allow for localhost only
    - 443:32443
    - 80:32080 # For Let's Encrypt

ArgoCD bootstraping

Please take a look this file addons-argocd-tpl.yaml to understand how we bootstrap the ArgoCD when the K3S started. Please note that this file is just only the template which the variables inside will be substituted when we call the start.bash script (see this file initial-argocd.sh for more detail).

Initial secrets

Because we don’t use vault in this demo, so we create the initial-secret Secret from the .env file instead and later use External Secret Operator to derive the applications secrets from it.