Skip to content

Gitlab CI/CD with Robot Framework#
Original Author Ville Purhonen
Checked By Juha-Matti Hietala
Updated By Alena Galysheva
Last Updated 27.03.2025

Introduction#

You can setup a Gitlab CI/CD pipeline to run automated tests with different triggers. This guide will walk you through a basic example and provide tips on how to make your setup more advanced if you're interested in exploring the DevOps aspect of test automation.

From Get started with GitLab CI/CD by GitLab:

Pipelines are made up of jobs and stages:

  • Stages define the order of execution. Typical stages might be build, test, and deploy.
  • Jobs specify the tasks to be performed in each stage. For example, a job can compile or test code.

Pipelines can be triggered by various events, like commits or merges, or can be on schedule. In your pipeline, you can integrate with a wide range of tools and platforms.

Runners are the agents that run your jobs and send results back to GitLab. These agents can run on physical machines or virtual instances. In your .gitlab-ci.yml file, you can specify a container image you want to use when running the job. The runner loads the image, clones your project, and runs the job either locally or in the container. If you use GitLab, runners on Linux, Windows, and macOS are already available for use. And you can register your own runners on GitLab.com if you’d like.

Setting up runner (optional)#

If you're interested, you can make a new runner and install it on a server. This is an optional step. Install GitLab runner on your server first, follow through these steps. Install Docker on the server if you haven't already.

After server installations are complete, you can proceed by making the runner in Gitlab. In the Gitlab project sidebar, go to Settings -> CI/CD. Under "Runners", click "New project runner". Tag the "Run untagged jobs" so you can test it initially. Make a Linux based runner as it is in default. Copy and paste the runner register info to notepad or something, it will only get displayed by a minute or two. Run the commands on your server, choose Docker as executor. Finally, run it with command gitlab-runner run.

Now you should have an operational GitLab runner. Its function is to automatically set up and run containers. This is how automatic testing and deploying services after code changes is usually done.

Setting up .gitlab-ci.yml file for running RF tests#

From Get started with GitLab CI/CD by GitLab:

To use GitLab CI/CD, you start with a .gitlab-ci.yml file at the root of your project. This file specifies the stages, jobs, and scripts to be executed during your CI/CD pipeline. It is a YAML file with its own custom syntax. In this file, you define variables, dependencies between jobs, and specify when and how each job should be executed. You can name this file anything you want, but .gitlab-ci.yml is the most common name.

The .gitlab-ci.yml file should be created in the root of the repository. The CI pipeline reads it and performs the steps specified in it. By default, pipeline is triggered on every commit. In order to run Robot Framework in a container, you should install all of the required dependencies in it. These include Python, pip package manager, Robot Framework itself, and third-party RF libraries. Web UI testing with Selenium requires browser drivers like geckodriver for Firefox and chromedriver for Chromium-based browsers. Browser library additionally requires NodeJS; the browser binaries are downloaded with this library and need to be installed with rfbrowser init command (see Browser library - Installation for more details).

You can find a curated list of Docker images to use for Robot Framework testing here.

Example setup#

.gitlab-ci.yml:

stages:
  - test

run-rf-tests:
  stage: test
  image: marketsquare/robotframework-browser
  script:
    - pip install -r requirements.txt
    - robot --outputdir results tests
  artifacts:
    paths:
      - results

This is a functional base. The example project follows the recommended RF project structure. Let's walk through what the script does.

The job called run-rf-tests contains a script section and belongs to the test stage. stage describes the sequential execution of jobs. A pipeline can have multiple stages (e.g., different stages for building an application, running tests against it, and for deploying it). If there are runners available, jobs in a single stage run in parallel. stage: test is used to specify that rf-run-tests job belongs to the test stage.

image is used to specify the container image to run the tests in. It's highly recommended to run your tests in a containerized environment to ensure you have control over the setup of the environment, configurations, dependency management, etc. marketsquare/robotframework-browser Docker image is used in this example. As mentioned in Robot Framework documentation, the image comes with the latest versions of robotframework-browser and robotframework packages and with pre-initialized browsers and other dependencies for running headful tests in the container.

script specifies the commands to execute. pip install -r requirements.txt handles the installation of the Robot Framework dependencies from the requirements.txt file located in the root of the project. The next line runs the tests and writes the output files to results directory (specified with --outputdir results option).

artifacts exports the files from the results directory (specified with paths: setting) as an artifact, which can be accessed in GitLab after the pipeline run is finished. To access them via GitLab UI, navigate to Build -> Artifacts and look at the icons on the right side. By default, the artifacts expire in 30 days. You can change this using the expire_in setting.

See CI/CD YAML syntax reference by GitLab for more information on the available configuration options.

Robot Framework setup#

Note: it's highly recommended to run the tests in headless mode in CI/CD environments.

Example .robot file using Browser library:

*** Settings ***
Documentation     Simple example using Browser library.
Library           Browser
Library           String


*** Variables ***
${LOGIN URL}          https://playwright.dev/
${BROWSER}            chromium
${IS_HEADLESS}        false
${BROWSER_OPTIONS}


*** Test Cases ***
Has Title
    New Browser    ${BROWSER}    headless=${IS_HEADLESS}
    New Page       ${LOGIN URL}
    Get Title      contains    Playwright

Example .robot file using Selenium library:

*** Settings ***
Documentation     Simple example using SeleniumLibrary.
Library           SeleniumLibrary
Library           String

Suite Setup       Set Browser Options

*** Variables ***
${LOGIN URL}          https://www.google.com/
${BROWSER}            Chrome
${IS_HEADLESS}        false
${BROWSER_OPTIONS}

*** Test Cases ***
Open Google Search Engine
    Open Browser    ${LOGIN URL}    ${BROWSER}    options=${BROWSER_OPTIONS}
    Title Should Be    Google
    [Teardown]    Close Browser

*** Keywords ***
Set Browser Options
    IF    $IS_HEADLESS == 'true'
        ${options}    Catenate    ${BROWSER_OPTIONS}    add_argument("--headless");
        VAR    ${BROWSER_OPTIONS}    ${options}    scope=SUITE
    END

Both of the examples expect the robot command to have the --variable IS_HEADLESS:true option set. Updated command looks like:

robot --outputdir results --variable IS_HEADLESS:true tests`

If you think that hard-coding all execution options in a .gitlab-ci.yml file is a bad idea, you're not alone. A more elegant way to handle management of the variables would include creating an argument file, listing robot options there, and passing it as a --argumentfile option.

Create a args folder in the root of the project and create a rf-ci.args file inside of it.

Add options and their values to the rf-ci.args, using a new line for each option. Our example rf-ci.args file would look like:

--outputdir results
--variable IS_HEADLESS:true

Now you can update .gitlab-ci.yml to run the RF tests with the following command:

robot --argumentfile args/rf-ci.args tests