How I develop the web application that syncs data from Google Drive (Deployed on Kubenetes)

This article is about to demonstrate the technique I use for the web application to retrieve the documents stored on Goolge Drive and make them publicly downloadable by web users.

My technique is not to make the web application to talk to Google Drive directly because it seems to be very complicate for the web application to do that. Instead, we will be using the Rclone utility to do that job, the Rclone will take the responsibility to copy files from Google Drive on every 5 minutes and the dowloaded files will be available for the web application to access. The web application will access those downloaded files like accessing the files on local file system.

Source code for this article can be found here https://github.com/its-knowledge-sharing/rclone-gdrive/blob/main/rclone-sync.yaml

Requirements

Last week my team got the requirements from the users, the requirements are…

• Customers can easily download the CoA documents via our website themselve.
• The CoA is the publicly downloadable documents so there is no need for the authentication to download them via website.
• The CoA documents are created by another team and stored them privately on the Google Drive
• The web application is deployed in Kubernetes

Technical Design

The direction of the arrows in the picture is the direction of data movement, not the direction of how connection is originated.

We will create a Kubernetes pod that wraps 2 containers inside, the 1st container is the NGINX (actually, this is the web application but I’m using NGINX for demonstration in this article). The 2nd container is the RClone that we will configure it to copy files from Google Drive.

The source code of Kubernetes Deployment can be found here on Github. Please note that on my production environment I always use Helm to render the YAMLs and use ArgoCD to deploy the application.

Rclone authentication to Google Drive

In order for Rclone to copy files from Google Drive, we will need to configure the authentication between them. The Rclone configuration file is similar to the snippet below. We use Google Workspace user token to authenticate to Google Drive instead of using a service account. Please focus on the this line token = <<Change-This>> for setting user token.

[GoogleDriveNAp]
type = drive
scope = drive.readonly
root_folder_id = <<Change-This>>
token = <<Change-This>>

For the ones who’re interested in configuring Rclone to authenticate Google Drive by using service account, please follow this youtube video — https://www.youtube.com/watch?v=t48_r_w64J0

The next question is, how to obtain the token value and populate it in the Rclone configuration file? This is what I did, I installed Rclone on my local Linux computer and run the rclone config command and follow the instruction here — https://rclone.org/drive/. Once we’re done with the rclone config command, finally we will see the user’s token on the screen so we can copy and paste it in the snippet above.

Another parameter that we need to configure is root_folder_id, this is ID of root Google Drive folder that we want Rclone to sync.

Please note that we need to grant folder Viewer permission for the user who own the token as shown in the pictures below.

Source Code

To simplify the application deployment on Kubernetes, I created a YAML file here. We won’t go too much in details about the meaning if the YAML, I’m assuming that we’re familiar enought with Kubernetes.

We need to alter the Secret rclone-config a little bit in order to create the right Rclone configuration file. Please change the root_folder_id and token field with the values obtained by the method I described in the earlier section.

apiVersion: v1
kind: Secret
metadata:
  name: rclone-config
stringData:
  rclone.conf: |
    [GoogleDriveNAp]
    type = drive
    scope = drive.readonly
    root_folder_id = <<Change-This>>
    token = <<Change-This>>

Also, if you’re going to use Ingress, please change the host of the Ingress to your own domain.

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: demo-genuine-doc
  labels:
    app.kubernetes.io/name: genuine-doc
    app.kubernetes.io/instance: demo
  annotations: 
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/force-ssl-redirect: "true"
spec:
  rules:
    - host: docs-demo.genuine-dev.napbiotec.io # <<< Change This Too
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: demo-genuine-doc
                port: 
                  name: http

Supports

Congratulation!!! if you’ve read the entire article and it is able to help you solve your issues. You can support me by:

• Follow me.
• Share my articles.
• Buy me a coffee via ADA address below if you want.