pabis.eu

Create and run pipelines with GitLab API

11 July 2023

Sometimes we have projects that build long or that we don't need builds for very often. We want to get the automated builds but only on demand. In GitLab we can set the job to manual. However, if we create jobs for each branch, the list of new pipelines will be very long, making it difficult to navigate. GitLab has a very powerful and extensive API. It almost allows you to write your own UI on top of it. Today, we will create a simple web UI that will allow users to launch triggered pipelines by selecting a branch or tag.

View the complete project at: https://github.com/ppabis/gitlab-cicd-ui

Getting the token

First we need to get the tokens for either a GitLab account, a group or a project. Here, we will use a key specific to the project. We head first to the project page, scroll down to settings, select Access Tokens.

Access Tokens

Create a new token with api permissions and Developer role. We will use this token to authenticate to GitLab. Save this token in a secure place.

Generate Token

List of branches and tags

Now as we have the token, we can start writing an app that will fetch the list of branches and tags. We will use Python and the requests library to make the API calls. Prepare the requirements.txt file and put just requests inside. Next install the dependencies with $ pip install -r requirements.txt.

In the new file branches.py we will write the code to fetch the list using the token provided in environment variables. The project ID will be hardcoded for now.

import requests, os
PROJECT_ID = 47513328 # Bring your own project ID
API_URL = "https://gitlab.com/api/v4/" # For self-hosted, change this
GITLAB_TOKEN = os.environ["GITLAB_TOKEN"] # Will fail if not set

def get_branches():
  url = f"{API_URL}/projects/{PROJECT_ID}/repository/branches"
  headers = {"PRIVATE-TOKEN": GITLAB_TOKEN}
  response = requests.get(url, headers=headers)
  return response.json()

In case we have a lot of branches, we need to paginate the results. GitLab API responses with x-next-page headers or HATEOAS links. We will use the links for simplicity as it is integrated in requests. Let's change our code to loop over several pages.

def get_branches():
  url = f"{API_URL}/projects/{PROJECT_ID}/repository/branches"
  headers = {"PRIVATE-TOKEN": GITLAB_TOKEN}
  response = requests.get(url, headers=headers)

  branches = response.json()
  iterations = 0
  # Loop at most 10 times
  while "next" in response.links and iterations < 10:
    response = requests.get(response.links["next"]["url"], headers=headers)
    branches += response.json()
    iterations += 1
  return branches

To test this, I created a script that will generate hundreds of branches in my repository.

for i in {1..300}; do
  RANDOM_STR=$(openssl rand -hex 6)
  git checkout -b issue/00$i-$RANDOM_STR
  echo $RANDOM_STR > branch_id.txt
  git add branch_id.txt
  git commit -m "Add branch id $RANDOM_STR"
  git checkout main
done

git push -u origin --all

Branches

Branches

By default GitLab will return 20 branches per page (also in the UI). We will use this limitation to see that not all branches will be loaded by our code because of the loop limit. Let's test this and see the size of the array we receive. At the end of file add a temporary code.

...
branches = len(get_branches())
print(f"Total branches fetched: {branches}")

The process takes some time to complete so it is better to have some caching in place but this is out of scope of this post.

$ python3 branches.py 
Total branches fetched: 220

This part is tagged as get-branches in the repository.*

Configuring the project's CI

Now that we have the list of branches, we need to configure the project's CI to run some jobs. We will use just fill the job definition with an echo and writing to file to have some artifacts. There will be special rules for the job to be created only on triggers rather than push or merge request. We can also specify some custom variables that will be passed to the job such as BUILD_TYPE.

stages:
  - build

build:
  stage: build
  tags: ["private"]
  script: |-
    echo "Current branch is ${CI_COMMIT_REF_NAME}"
    echo "Hello world from ${CI_COMMIT_REF_NAME}" > hello.txt
    echo "Requested BUILD_TYPE is ${BUILD_TYPE}"
    echo "Build type => ${BUILD_TYPE}" >> hello.txt
  rules:
    - if: '$CI_PIPELINE_SOURCE == "trigger"'
  artifacts:
    paths:
      - hello.txt

Listing the pipelines

Next we can list some of the most recent triggered pipelines. We will query the API for just the last 50 pipelines.

def get_pipelines():
    url = f"{API_URL}/projects/{PROJECT_ID}/pipelines?per_page=50"
    headers = {"PRIVATE-TOKEN": GITLAB_TOKEN}
    return requests.get(url, headers=headers).json()

This part is tagged as get-pipelines in the repository.

Triggering a pipeline

Now we can create a function that will trigger a pipeline for a specific branch and BUILD_TYPE.

def trigger_pipeline(branch, build_type):
    url = f"{API_URL}/projects/{PROJECT_ID}/trigger/pipeline"
    headers = {
        "Content-Type": "application/x-www-form-urlencoded"
    }
    body = {
        "token": TRIGGER_TOKEN,
        "ref": branch,
        "variables[BUILD_TYPE]": build_type
    }
    response = requests.post(url, headers=headers, data=body)
    return (response.status_code, response.json())

For that we will need a trigger token. Go to the project's settings and create a token under CI/CD.

Trigger token

This part is tagged as trigger-pipeline in the repository.

Web UI

Next we will pack it into a web UI form so we can use it. I will use built in Python HTTP server but feel free to experiment with frameworks.

First I will create a string template directly in the code. It will have {} to fill the blanks during index generation.

INDEX_HTML = """
<!DOCTYPE html>
<html>
<head>
  <title>GitLab Project CI</title>
  <!-- Bootstrap CSS -->
  <link href="https://cdn.jsdelivr.net/npm/bootstrap@5.3.0/dist/css/bootstrap.min.css" rel="stylesheet" integrity="sha384-9ndCyUaIbzAi2FUVXJi0CjmCapSmO7SnpJef0486qhLnuZ2cdeRhO02iuK6FUUVM" crossorigin="anonymous">
</head>
<body>
  <h2>Run Pipeline</h2>
  <form action="/" method="POST">
    Branch: <select name="branch">
      {}
    </select>
    <br>
    Build type:<select name="build_type">
      <option value="debug">Debug</option>
      <option value="release">Release</option>
    </select>
    <br>
    <input type="submit" value="Submit">
  </form>
  <h2>Recent pipelines</h2>
  <table>
    {}
  </table>
</body>
</html>
"""

Next I will implement a function that will render index page based on this template.

def format_index():
    branches = [f"<option value=\"{b['name']}\">{b['name']}</option>" for b in get_branches()]
    html_branches = "\n".join(branches)

    pipelines = [
        f"<tr><td><a href=\"{p['web_url']}\">{p['id']}</a></td><td>{p['ref']}</td><td>{p['status']}</td></tr>"
        for p in get_pipelines()
    ]
    html_pipelines = "\n".join(pipelines)

    return INDEX_HTML.format(html_branches, html_pipelines)

Also we would like to have HTTP handler that will react to GET requests for showing the web UI and to POSTs when the user wants to trigger a pipeline.

class GitLabHandler(BaseHTTPRequestHandler):
    def do_GET(self):
        self.send_response(200)
        self.send_header("Content-type", "text/html")
        self.end_headers()
        self.wfile.write(format_index().encode("utf-8"))
        self.wfile.close()

    def do_POST(self):
        content_length = int(self.headers["Content-Length"])
        post_data = self.rfile.read(content_length).decode("utf-8")
        parsed_data = parse_qs(post_data)
        branch = parsed_data["branch"][0]
        build_type = parsed_data["build_type"][0]
        print(f"Triggering pipeline for branch {branch} with build type {build_type}")
        code, data = trigger_pipeline(branch, build_type)
        if code >= 400:
            self.send_response(400)
            self.send_header("Content-type", "application/json")
            self.end_headers()
            self.wfile.write(json.dumps(data, indent=2).encode("utf-8"))
        else:
            self.send_response(302)
            self.send_header("Content-type", "text/html")
            self.send_header("Location", "/")
            self.end_headers()
            self.wfile.write("Redirecting".encode("utf-8"))
        self.wfile.close()

Lastly what we need is to start the server and attach the new handler class to it. Moreover, I listed also all the required imports below.

# ...
from http.server import BaseHTTPRequestHandler, HTTPServer
from urllib.parse import parse_qs
import json
# ...

def main():
    server_address = ("", 8000)
    server = HTTPServer(server_address, GitLabHandler)
    print("Starting server")
    server.serve_forever()

if __name__ == "__main__":
    main()

The end effect of the UI looks like this:

Form for running the pipeline

However, currently there are no pipelines that were triggered. We need to commit the .gitlab-ci.yml file to our target project. We will also need to merge this changes into all the branches that we want to be able to trigger the pipeline with.

$ branches=$(git for-each-ref --shell --format="%(refname:short)" refs/heads/ | tr -d "'")
$ for branch in $branches; do
$     git checkout $branch
$     git merge -m "merge CI file" main
$ done
$ git push -a

As a side note to running pipelines on GitLab.com, you need to create a runner on your own machine and register it or provide payment details to be able to use GitLab managed runners. On self-hosted GitLab you just need to create a runner. In the .gitlab-ci.yml I specified tag for my runner. Also I am assuming that you will use docker executor.

Registering the runners

This part is tagged as web-ui in the repository.

Running the pipeline

Finally, we can run the pipeline. Select the build type and branch and try submitting the form.

Form for running the pipeline

Follow the link to GitLab to see how the job finishes and look at the artifacts. It seems that our variables were passed correctly.

Job artifact