Argo CD with Gitlab CI - Part I

According to the official documentation Argo CD is a declarative, GitOps continuous delivery tool for Kubernetes. It monitors the live state of Kubernetes clusters against the single point of truth in git configuration repositories. Gitlab with their continuous integration offering allows management of source code and Kubernetes configurations used by Argo CD.

• Argo CD with Gitlab CI - Part I (this article)

  In this article we will outline an example workflow how to integrate Gitlab CI with Argo CD to achieve a robust SDLC¹ process that is suitable for enterprises as well as for your Homelab. This article will focus on the CI part with Gitlab.

• Argo CD with Gitlab CI - Part II

  Goes into details how to deploy and use ArgoCD. It basically leverages the state and artifacts (docker images) generated and maintained via Gitlab.

Assumptions in this series

• Gitlab instance URL
  • gitlab.yourdomain.com
• Source code repository (called ‘Git Source Repo’ in diagram)
  • smart-home/lighting (full URL: gitlab.yourdomain.com/smart-home/lighting)
• Kubernetes configuration repository (called ‘Git Config Repo’ in diagram)
  • deployments/lighting (full URL: gitlab.yourdomain.com/deployments/lighting)
• Registry URL for docker images (part of Gitlab, but can be any docker registry)
  • registry.yourdomain.com
• Deployment configurations
  • Kustomize is used as framework for Kubernetes configuration management. The process is the same for raw kube config files or helm charts.

The Process

The following sequence diagram outlines different stakeholders and their role in different phases of the SDLC. If you are reading that as a one-man-shop or smaller company, e.g., to deploy those processes in your Homelab or startup you most likely will (A) skip some of the merge requests and (B) setup your CI pipelines to automatically close the merge requests when the pipeline succeeds. Closing merge requests to the configuration repositories is also useful for automated deployments to environments such as development or even QA/staging.

More specifically for point (A) that means there is no QA Engineer reviewing the code, but the code will be directly pushed into the respective main branch and/or the merge request will be closed by the Software Engineer itself.

For point (B) that means that there will be no Release Engineer, but the merge request in the Git Config Repos will be closed automatically.

The following sequence diagram shows what happens when a Software Engineer pushes new code to the source code repository. This flows should be adapted by you according to your needs and according to the environments you have in your setup.

Some example flow to get you thinking:

• Each commit triggers the flow below, but automatically closes the version change merge request and if successful deploys to your remote development environment.
• For releases, only triggered for tags, the version change merge request is reviewed and closed by a Release Engineer, before it is deployed to a pre-production or staging environment.
• Production is manually triggered by running the same deployment pipeline as in the previous step, if the release is qualified.

Setting up Gitlab repositories with CI

For the sake of separation of concerns and to simplify and secure the management of different parts of the process we create two different git repositories. One is the software repository that includes the source code of your application that will be deployed, the second one is the configuration repository including all the Kubernetes config files and version information. The latter is used by Argo CD to deploy the application to one or more environments and to make sure the environments has always the correct state.

Source Code Repository and .gitlab-ci.yml

As example we create a repository lighting under the group smart-home, resulting in the full repository path as smart-home/lighting. This repository includes the following .gitlab-ci.yml file, defining the CI build and trigger of version change jobs.

stages:
  - test
  - build
  - deploy

# Run with production runner in production k8s cluster
default:
  tags:
    - production

include:
  - template: Security/SAST.gitlab-ci.yml

######################################################################
# PRODUCTION
docker build main:
  stage: build
  needs: []
  image:
    name: gcr.io/kaniko-project/executor:debug
    entrypoint: ['']
  script:
    - cat ${INTERNAL_ROOT_CA_FILE} >> /kaniko/ssl/certs/additional-ca-cert-bundle.crt
    - echo "{\"auths\":{\"$CI_REGISTRY\":{\"username\":\"$CI_REGISTRY_USER\",\"password\":\"$CI_REGISTRY_PASSWORD\"}}}" > /kaniko/.docker/config.json
    - /kaniko/executor --context $CI_PROJECT_DIR --dockerfile $CI_PROJECT_DIR/Dockerfile --destination $CI_REGISTRY_IMAGE
  only:
    - main

update deploy config:
  stage: deploy
  needs: ['docker build main']
  trigger:
    project: deployments/${CI_PROJECT_NAME}
    branch: main
  variables:
    BUILD_VERSION: ${CI_COMMIT_SHORT_SHA}
    IMAGE_TAG: latest
    DOCKER_IMAGE: ${CI_REGISTRY_IMAGE}
    # needs to match the overlays directory in the downstream repository
    ENVIRONMENT: production
  only:
    - main
######################################################################

Some notes to this .gitlab-ci.yml:

• The production tag is used to assure that the jobs are run by a gitlab runner that is marked for production. This is completely optional and dependence on your setup.
• The SAST scanning is optional too, but strongly recommended to find potential security flaws in your software.
• The INTERNAL_ROOT_CA_FILE variable is defined as Gitlab instance environment variable and includes a self-signed Root CA. This is necessary because the repository is hosted on a private Gitlab instance with a self-signed certificate. If you are hosting your source code repository on gitlab.com or you have signed certificate from one of the well known root CA you don’t need this line.
• update deploy config triggers a down stream task in our configuration repository, see next section. The project name in this example is lighting and needs to match between the two repositories. This is not a requirement, but allows the writing of more generic .gitlab-ci.yml files and organizes the repositories in a more structured way.

Configuration Repository with .gitlab-ci.yml

Based on the repository name and .gitlab-ci.yml file from the section above we need to create the configuration repository under deployments/lighting. If you want to change the group name and repository name make sure to update the .gitlab-ci.yml file of your source code repository.

stages:
  - test
  - update

verify kustomize base:
  stage: test
  image: registry.yourdomain.com/deployments/helper:latest
  script:
    - kustomize build k8s/base

verify kustomize production:
  stage: test
  image: registry.yourdomain.com/deployments/helper:latest
  script:
    - kustomize build k8s/overlays/production

include:
  - project: 'deployments/helper'
    ref: main
    file: '/templates/.gitlab-ci-template.yml'

As you can see we are verifying if our kustomize files are valid as part of each CI run. Argo CD supports also plain kubectl config files and helm charts. The main logic is imported from a template .gitlab-ci-template.yml file in the deployments/helper repository. This allows us to reuse the same logic for all, or at least most, of our deployments without copy & pasting code.

The content of the deployments/helper/templates.gitlab-ci-template.yml file is shown below.

stages:
  - update
update version:
  stage: update
  image: registry.yourdomain.com/deployments/helper:latest
  before_script:
    - cat ${INTERNAL_ROOT_CA_FILE} >> /tmp/internal-ca-cert.crt
    - git config http.sslCAInfo /tmp/internal-ca-cert.crt
    - git config user.email "[email protected]"
    - git config user.name "Gitlab CI Job"
    # Access token with 'write_repository' permissions and 'maintainer' role,
    # see group configuration 'gitlab-ci-version-update' token and CI_ACCESS_TOKEN env variable
    - git remote set-url origin https://gitlab-ci-token:${CI_ACCESS_TOKEN}@${CI_SERVER_HOST}/${CI_PROJECT_PATH}.git
  script:
    - export FEATURE_BRANCH=${ENVIRONMENT}-${IMAGE_TAG}-${BUILD_VERSION}
    - git checkout -b ${FEATURE_BRANCH}
    - (cd k8s/overlays/${ENVIRONMENT} && kustomize edit set image ${DOCKER_IMAGE}:${IMAGE_TAG})
    - export COMMIT=https://$(echo -n ${DOCKER_IMAGE} | sed 's/registry/gitlab/g')/-/commit/${BUILD_VERSION}
    - export CURRENT_BUILD=$(yq e '.commonAnnotations.build' k8s/overlays/${ENVIRONMENT}/kustomization.yaml)
    - export CHANGELOG=https://$(echo -n ${DOCKER_IMAGE} | sed 's/registry/gitlab/g')/-/compare/${CURRENT_BUILD}...${BUILD_VERSION}
    - (cd k8s/overlays/${ENVIRONMENT} && kustomize edit set annotation build:${BUILD_VERSION} commit:${COMMIT} changelog:${CHANGELOG})
    - git add k8s/overlays/${ENVIRONMENT}
    - git diff HEAD
    - kustomize build k8s/overlays/${ENVIRONMENT}
    - git commit -m "Updating environment '${ENVIRONMENT}' with build '${BUILD_VERSION}' and tag '${IMAGE_TAG}'" || exit 0
    - if [ "$AUTOMERGE_MR" = true ] ; then AUTOMERGE="-o merge_request.merge_when_pipeline_succeeds"; fi
    - git push origin ${FEATURE_BRANCH}
        -o merge_request.create
        -o merge_request.target=${CI_COMMIT_BRANCH}
        -o merge_request.remove_source_branch
        -o merge_request.title="RELEASE - ${ENVIRONMENT} release of build ${BUILD_VERSION} with tag ${IMAGE_TAG}"
        -o merge_request.description="Automatically created by Gitlab CI for ArgoCD controlled releases."
        ${AUTOMERGE}
  rules:
    - if: $BUILD_VERSION && $ENVIRONMENT && $DOCKER_IMAGE && $IMAGE_TAG && $CI_COMMIT_BRANCH

This is the secret sauce that connects our CI phase with our CD phase. It gives us full traceability by following GitOps principles. Changes are committed directly by the CI job. Those changes are committed in a new feature branch and only merged if AUTOMERGE_MR is set to true. This flag is passed on via the source code repository CI job. In our example it is not passed on, see first .gitlab-ci.yml file, defaulting to false. Furthermore the merge request CI pipeline needs to pass in order for the changes to be merged.

Everything else in this .gitlab-ci.yml file modifies, commits and pushes version changes and related meta data. Making all the change explicit and giving Argo CD the configuration files in plain text. This makes the whole process also auditable by having not only dedicated commits, but also merge requests.

Continue reading with part 2 of the series

Appendix: Used CI Image

The following Dockerfile is used to build the image registry.yourdomain.com/deployments/helper:latest used in the CI jobs outlined above. It is build and pushed to registry.yourdomain.com from the Gitlab repository deployments/helper. The same repository that also includes the Gitlab CI template file.

FROM alpine:3.16

RUN apk add --no-cache git curl bash yq
RUN (cd /usr/bin && curl -s "https://raw.githubusercontent.com/kubernetes-sigs/kustomize/master/hack/install_kustomize.sh"  | bash)

If you are using helm or any other tools during the deployment install them accordingly. Also make sure that the alpine version is up to date.