Application Logs using Grafana Loki on GKE

Loki is a log aggregation system designed to store and query logs from all applications. In this article we will use Google Cloud Storage (GCS) as the storage of Loki and the Loki itself will be deployed on Google Kubernetes Engine (GKE).

Introduction

This is one another requirement that I got from the customer after I’ve done the Cortex setup that I wrote in the previous article here. My customer wants to have a single place to store the application logs (containers logs) from many Google Compute Engine (GCE) instances. Not just only to store the logs, he also wants to query and aggregate those logs, use GCE instance name as the key index for the query.

By reading the comparisons here, I then made the decision to use Loki instead of Elasticsearch. Very similar to the previous Cortex article, these are the tech stack that we will use.

• Google Kubernetes Engine (GKE) — Kubernetes cluster on Google Cloud Platform that we will deploy the workloads.
• Google Cloud Storage (GCS) — The actual storage that we will configure Cortex to use it as the storage.
• Loki — Data source to store logs.
• Grafana — The web user interface for log querying.
• Promtail — The agent that ships the contents of logs to a private Grafana Loki. In this case, the log contents are from containers run on the GCE instances.

Source Code

The source codes relate to this article are kept publicly in the GitHub repository here — https://github.com/its-knowledge-sharing/setup-loki-gke. I would recommend to clone the code and read them at the same time you read this article.

Creating the GKE cluster

I’m assuming that we are familiar to the Google Cloud Platform (GCP) and already have the GCP account.

gcloud container clusters create gke-loki --zone us-central1-a --project ${PROJECT}

Run the gcloud command above and wait for a while until the GKE cluster is created. Also, use command below to create the KUBECONFIG file to authenticate to our GKE cluster.

gcloud container clusters get-credentials gke-loki --zone us-central1-a --project ${PROJECT}

Creating the GCS — Cloud storage for Loki

GCS is the object storage of GCP, it is comparable to the S3 of Amazon Web Services (AWS). The reason why we use GCS to store logs from Promtail instead of using traditional file system because the maintenance operation is easier (diskspace expansion). The cost of GCS is also cheaper than the traditional file system when compare to the same size.

To create the GCS bucket, run the command below. In this case I keep the environment variables I need in the .env file and export it in my own bash script. Please note that we create the GCS bucket in the same region as the GKE cluster, which is us-central1 in this case.

#!/bin/bash
source .env
gsutil mb -b on -l us-central1 -p ${PROJECT} gs://${BUCKET_NAME}/
gsutil ls gs://${BUCKET_NAME}/

Creating service account for Loki to access GCS

In order to allow Loki able to write/read data from GCS bucket, we will need to create the IAM service account and will configure Loki to use it at runtime.

Run the command below to create the IAM service account (SA).

gcloud iam service-accounts create ${SA_NAME} --project ${PROJECT} --display-name="Service account for Loki"

Once the SA is created, we will need to assign the role what this SA can do. For the sake of simplicity, we will grant it with the role storage.objectAdmin. Please keep in mind that in the real life we should limit the access to only read/write and specific to the bucket.

gcloud projects add-iam-policy-binding ${PROJECT} \
--member="serviceAccount:${SA_NAME}@${PROJECT}.iam.gserviceaccount.com" \
--project ${PROJECT} \
--role="roles/storage.objectAdmin"

Deploying Loki to GKE

The easiest way we deploy the application (Loki in this case) into Kubernetes is by using Helm. Loki also provided it’s own Helm chart here https://grafana.github.io/helm-charts/. What we need to do is to create the Helm values file to customize what we need and run some Helm commands to get everything.

To simplify things, I wrote a Loki deployment script which can be cloned from this GitHub repository — https://github.com/its-knowledge-sharing/setup-loki-gke/blob/main/04-deploy-loki.bash.

The code snippet below excerpted from from the script mentioned above. We need to download service account key file (JSON format) first. Then we will create the Kubernetes Secret resource from file we downloaded.

# Create service account secret
gcloud iam service-accounts keys create ${KEY_FILE} --iam-account=${SA}
kubectl delete secret ${SECRET} -n ${NS}
kubectl create secret generic ${SECRET} --from-file=gcp-sa-file=${KEY_FILE} -n ${NS}

Once the secret is created, we should be able to see the secret gcp-sa by running “kubectl” command shown below. Later this secret will be mounted into Loki pods which will be used for authentication to GCP.

$ kubectl get secret -n loki gcp-sa
NAME TYPE DATA AGE
gcp-sa Opaque 1 16m

Now it’s the time to deploy Loki into “loki” namespace by using Helm as the template engine. The Kubernetes manifest file is written into temp file “tmp-loki.yaml” which will be later used by “kubectl”.

helm repo add loki-helm https://grafana.github.io/helm-charts/
helm template loki loki-helm/loki-distributed \
-f loki/loki.yaml \
-f loki/loki-volume.yaml \
--set customParams.gcsBucket=${BUCKET_NAME} \
--version 0.45.1 \
--namespace ${NS} > tmp-loki.yaml
kubectl apply -n ${NS} -f tmp-loki.yaml

Please note that we have 2 Helms values files, the loki-volume.yaml is the one contains the configuration for service account secret volume mounting.

Please see Loki Helm values file here — loki.yaml and loki-volume.yaml for better understanding.

If everything is OK, we should be able to see the pods similarly shown as in the picture below.

kubectl get pods -n loki

kubectl get svc -n loki

In order to make Promtail from outside the GKE cluster able to send the logs to Loki, we will need to create Kubernetes ingress resource. The Kubernetes ingress resource on GKE will be internally used to create GCP HTTP Load balance.

The correspond YAML to create the ingress is — https://github.com/its-knowledge-sharing/setup-loki-gke/blob/main/loki/loki-ing.yaml. Please note that the ingress routes the traffic to Loki via loki-log-gateway ClusterIP service.

kubectl apply -n ${NS} -f loki/loki-ing.yaml

Once the ingress is created, we will see the ingress resource as shown in the picture below. Later we will use the IP address in the “ADDRESS” column when we setup the Promtail.

kubectl get ing -n loki

Deploying Grafana to GKE

The Loki is just the data source that store the data, The easiest way to view those data is by using Grafana. Similar to the Loki we deployed earlier, we will need to deploy the Grafana by using Helm too.

The script to deploy Grafana and it’s ingress can be found here — https://github.com/its-knowledge-sharing/setup-loki-gke/blob/main/05-deploy-grafana.bash.

helm repo add grafana-helm https://grafana.github.io/helm-charts
helm template grafana grafana-helm/grafana \
-f grafana/grafana.yaml \
--skip-tests \
--namespace ${NS} > tmp-grafana-loki.yaml
kubectl apply -n ${NS} -f tmp-grafana-loki.yaml
kubectl apply -n ${NS} -f grafana/grafana-ing.yaml

The Helm values file to configure the Grafana also found here — https://github.com/its-knowledge-sharing/setup-loki-gke/blob/main/grafana/grafana.yaml. if everything is OK, we should be able to see the ingress and Grafana as shown below.

kubectl get ing -n grafana-loki

Please note that we should see the “Loki” data source in the “Explorer” menu

Use this command to get the default Grafana default password. Keep in mind that the password is changed every time if pod is restarted.

kubectl get secret grafana-loki \
-n grafana-loki \
-o jsonpath="{.data.admin-password}" | base64 --decode

Create GCE instance

Now this is the time to create the GCE instance and deploy Node Exporter and Promtail. We will deploy Node Exporter just for using it’s logs to send to Loki by using Promtail.

gcloud compute instances create promtail-001 \
--image=projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20220308 \
--image-project=${PROJECT} \
--machine-type=projects/its-artifact-commons/zones/us-central1-a/machineTypes/e2-medium \
--zone=us-central1-a

Once the GCE is running, we then need to install the docker and docker-compose. We will run Promtail and Node Exporter via docker-compose. Doing this will make our life a lot easier. Please use the code below to install the required components, run this code in the GCE instane.

#!/bin/bash
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -
sudo add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable"
sudo apt-get -y update
sudo apt-get -y install docker-ce docker-ce-cli containerd.io
sudo systemctl enable docker
# install docker-compose following the guide: https://docs.docker.com/compose/install/
sudo curl -L
"https://github.com/docker/compose/releases/download/1.25.4/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose
sudo chmod +x /usr/local/bin/docker-compose

Start docker-compose by using docker-compose.yaml here — https://github.com/its-knowledge-sharing/setup-loki-gke/blob/main/docker-compose.yaml.

We also created another wrapper script here to start docker-compose shown in the snippet below.

#!/bin/bash
DATA_DIR=$(pwd)
LOKI_DOMAIN=<change-this>
INSTANCE=$(hostname)
ENV_FILE=.env
sudo cat << EOF > ${ENV_FILE}
DATA_DIR=${DATA_DIR}
INSTANCE=${INSTANCE}
CONTAINERS_LOG_DIR=/var/lib/docker/containers
LOKI_DOMAIN=${LOKI_DOMAIN}
REGION=us-central1
ZONE=us-central1-a
GROUP=demo
EOF
sudo docker-compose up -d --remove-orphans

Once docker-compose is started, the logs should be automatically sent to Loki that’s running on GKE.

Verifying the result

We should see the logs in the Grafana “Explore” menu. Try using LogQL below to see if we can see the logs or not.

{host=”promtail-001", container_name=”promtail”}