Tutorial
7 min read

How we helped our client to transfer legacy pipeline to modern one using GitLab's CI/CD - Part 2

Please dive in the second part of a blog series based on a project delivered for one of our clients. If you miss the first part, please check it here.

Table of Contents

PART I
  • Problem description
  • General description of the solution
  • Problem 1: Limited job output size in GitLab
  • Problem 2: Limited duration of jobs running on shared runners
PART II
PART III
  • Problem 7: Passing on artifacts between CI/CD jobs
  • Problem 8: Starting docker build manually
  • Problem 9: We cannot rely on the error code returned by Puppet
  • Summary



Problem 3: Building a container image in the job

In several of the CI/CD jobs in our project, we create container images and upload them to GitLab's Registry. Building container images in GitLab's CI/CD requires infrastructure preparation, or we have to use a tool other than Docker. If we want to build images using Docker, then we must give the Docker client access to the Docker daemon socket, which is not recommended for security reasons. If we want to use shared GitLab.com runners, access to the daemon socket is not possible.

We can solve the problem in several ways:

  • Using an executor in a runner that does not work in a container and which has access to a Docker socket (e.g. shell executor). In this case, we can't use GKE or shared GitLab.com runners. We need to install, configure and maintain such a runner ourselves.
  • Using Kubernetes or Docker executor, but with Docker socket mounted inside containers. The solution requires an independent runner, prevents the use of shared runners and is bad from a security perspective.
  • Using the buildah tool to build the container image inside the container and the skopeo program to upload the image to the GitLab Registry. Both programs are part of the https://github.com/containers project and do not require a privileged daemon norrootaccess (unlike Docker).
  • Use kaniko. Kaniko is a tool dedicated to building container images in Kubernetes and in containers. Like buildah, it doesn't require special permissions. An example of its use is described in the official GitLab documentation.

Our team has already had experience with Kaniko in other projects, so we made use of it in this project.

The definition of a job using kaniko in the .gitlab-ci.yml looks like this:

build-base-image:
  stage: prepare
  image:
    name: gcr.io/kaniko-project/executor:debug
    entrypoint: [""]
  script:
    - echo 
"{\"auths\":{\"$CI_REGISTRY\":{\"username\":\"$CI_REGISTRY_USER\",\"password\":\"$CI_REGISTRY_PASSWORD\"}}}" > /kaniko/.docker/config.json
    - /kaniko/executor --context $CI_PROJECT_DIR --dockerfile 
    $CI_PROJECT_DIR/Dockerfile --destination 
    $CI_REGISTRY_IMAGE:$CI_COMMIT_TAG

As you can guess, after studying the above code fragment, Kaniko builds a container image and immediately uploads it to the indicated image registry.

Problem 4: The GitLab Registry token expires too quickly

GitLab has an integrated container image registry (Registry). A one-time password is created for each job, which allows you to use Registry without having to manually create a dedicated account. GitLab passes the login and temporary password to the Runner, which then sets them in the environment variables of the job process.

This is the end result of a successful job output that creates a container image using Kaniko and uploads it to the Registry built into GitLab:

null

In our project, some jobs take a very long time, longer than the lifetime of a one-time Registry password. Limiting the duration of such a one-time password is necessary from the security point of view.

Uploading the image to the Registry fails and ends with the following error:

null

We solved this problem in a traditional way:

  • We created a dedicated technical account in GitLab.
  • We gave the new account project privileges.
  • We logged in to GitLab on this account and created the personal access token. This token doesn't have an expiration date.
  • In the project, in the CI/CD settings, we created two new variables containing the login and password (token) of the technical account respectively
  • We modified the job so that Kaniko used a dedicated technical account to perform operations in the Registry.

After these modifications, we no longer had problems uploading container images to the Registry, even if the CI/CD job took 3 hours.

Problem 5: In the paid GitLab.com plan we have a limit on the shared runners used time

If you use paid GitLab-as-a-service plans, you can use a certain number of minutes for the CI pipeline, i.e. shared Runners. Each of the paid plans has a different limit. Using shared Runners is very convenient because we don't have to worry about maintaining our own Runners and thus save time and money. The limit may be sufficient in some projects, but in our project we quickly reached the monthly limit.

You will see this type of message when you use the available time of shared Runners in a given month:

null

We solved this problem by setting up a dedicated Kubernetes cluster for Runners. We allow GitLab to decide which Runner to use (shared or our own). Thanks to this, the load is distributed to both types of Runners, we reduce expenses and shorten the time of pipeline execution.

Instructions for using Kubernetes cluster as the platform for Runners are described in the Problem 2 section.

Problem 6: User's names with national characters in GitLab

If the runner or container we use in our CI/CD has locales that do not support UTF-8, and the user making commits to the repository that has characters that are not ASCII in the name or surname, then the CI/CD job may end with the following error:

FAILURE: Build failed with an exception.

* What went wrong:
Could not set the value of environment variable 'GITLAB_USER_NAME': 
could not convert string to current locale

There is even an issue with this in this GitLab project.

A workaround is also provided there. You can change the value of the GITLAB_USER_NAME variable so that it doesn't contain non-ASCII characters. It can be assigned, for example, to the value of the variable containing the user's login (assuming that the login consists only of ASCII characters).

To the before_script section in .gitlab-ci.yml, add:

  # Workaround for "Could not set the value of environment variable 
'GITLAB_USER_NAME': could not convert string to current locale" problem.
  # https://gitlab.com/gitlab-org/gitlab-foss/issues/38698
  - export GITLAB_USER_NAME=$(echo $GITLAB_USER_LOGIN)

Next part of the post

Follow our profile on Linkedin and stay up to date for the next part!

big data
kubernetes
google cloud platform
cloud
CI/CD
11 August 2020

Want more? Check our articles

5 reasons to follow us on Linkedin. Celebrating 1,000 followers on our profile!

We are excited to announce that we recently hit the 1,000+ followers on our profile on Linkedin. We would like to send a special THANK YOU :) to…

Read more
Use-cases/Project

Anomaly detection implemented in podcasting company

Being a Data Engineer is not only about moving the data but also about extracting value from it. Read an article on how we implemented anomalies…

Read more
Tutorial

Apache NiFi - why do data engineers love it and hate it at the same time? Blog Series Introduction

Learning new technologies is like falling in love. At the beginning, you enjoy it totally and it is like wearing pink glasses that prevent you from…

Read more
Use-cases/Project

Business value of event processing - use cases

Every second your IT systems exchange millions of messages. This information flow includes technical messages about opening a form on your website…

Read more
Big Data Event

Big Data Tech Warsaw Summit 2019 summary

It’s been already more than a month after Big Data Tech Warsaw Summit 2019, but it’s spirit is still among us — that’s why we’ve decided to prolong it…

Read more
Tech News

Celebrating GetinData’s Inclusion on Clutch’s Lists of Top Big Data and IoT Companies!

Founded by former Spotify data engineers in 2014, GetInData consists of a team of experienced and passionate Big Data veterans with proven track of…

Read more

Contact us

Fill out this simple form. Our team will contact you promptly to discuss the next steps.

hello@getindata.comFist bump illustration

Any questions?

Choose one
By submitting this form, you agree to our  Terms & Conditions