Overview

Structure

The Qt Continuous Integration project (CI) is a collection of scripts that work together to provide a complete continuous integration environment for projects with complex dependencies.

At the heart of the CI is the scheduler. It receives events (such as a commit staged in gerrit) and processes them. Dependencies are deduced by the dependency resolver which is queried by the scheduler in order to build a set of work items. Let’s assume module B should be tested and it depends on module A. The dependency resolver will maintain git checkouts of all modules. It then looks at the ‘dependencies.yaml’ in the root or coin directory of B and from there sees that A is required as dependency. The scheduler then uses this information to schedule a build of A, a build of B which waits for A to complete, and a test run for B.

The system is build with artifacts in mind, it doesn’t just build a module over and over. Instead, when we ask to test module B again, it will see that there are already build artifacts for module A. Unless A changed of course. But assuming the system is used to control the repository of A as well, it will always have an up to date build of A. The storage module is responsible for managing the artifacts and build logs.

At this point in time the system is tightly coupled to gerrit and git as source control systems.

The scripts that build Coin discover each other using a simple RPC name service, which is bootstrapped using an environment variable or a command line parameter.

The scripts communicate with each other using Apache Thrift over TCP sockets as RPC mechanism.

Interaction with Gerrit

Coin interacts with gerrit in two ways. Firstly the changes are polled periodically and Coin will create integrations from staged commits. Then the changes move to integrating state, which tells that coin is working on them. After that commits are either merged or rejected and returned to new state.

Coin also listens to gerrit event stream, this is where coin will pick up the precheck events. The default precheck configurations are listed in the precheck.yaml in each product repo, in most cases the product is qt5. See platform configurations for details.

Precheck can also define the platforms as a list and Coin will parse the platforms based on that.

Coin might attach various info into the gerrit message after integrations and prechecks. This info can include info about test that failed e.g. flaky metrics. Some Anomalies that happened during the run are shown, particularly ones that can relate to changes.

Debugging

To get more verbose debug output, use the –verbose option for each script. Alternatively you can export an environment variable before running any of the scripts:

export QT_CI_DEBUG=*

to only increase debug output in one area:

export QT_CI_DEBUG=CI.Scheduler

or several areas:

export QT_CI_DEBUG="CI.Scheduler CI.Application CI.Storage"

Vocabulary

Term	Definition
CI	Continuous Integration
agent	A runner for some workitem. Identified by host name and port.
project	namespace/project_name: A git module that is built and tested with the CI. For example each of Qt’s git modules is a project. A project must always include its namespace. Examples are “qt/qtbase”, “qt/qtdeclarative”, it is always a string.
change	gerrit_instance, project, ref, branch: A change is a patch in a project. It is defined by project and sha1. To be useful in the context of the CI, each change has a branch associated. The branch is the target when testing a change. For testing each change will be cherry-picked on top of it’s target branch and if found good, it is cherry-picked into the project’s git repository. In addition each change keeps a reference to which gerrit instance it belongs.
buildKey	project/tree-sha1/test_configuration_key/dependency_hash/request_type: A string that represents the build or testing of a change. It encodes the change including dependencies (a hash of all dependencies’ sha1s).
baseKey	project/tree-sha1/test_configuration_key/dependency_hash: This is provided for convenience and represents the part of ‘buildKey’ that is shared between different ‘WorkItem’ types with identical dependency paths.
WorkItem	A task internal to the scheduler. It is either a provision, build, test or final item. Build items represent the building of one change, test items running the tests for a change and final items collect results from all items needed to approve or disapprove a change.

Workitems

Coin implicitly creates dependencies for workitems from provisioning to build, and from build to test. A workitem might run one or more VMs if some retry condition is hit, these are specified in more detail below for each type. Every VM for same workitems runs exactly the same set of commands.

If a retry happens after the VM has started an anomaly is recorded. This anomaly will be seen in the API as the workitem property and posted to gerrit in the message resulting from the run.

Provisioning

Provisioning creates a new VM image for all subsequent build and test workitems to use. Provisioning requires a verify boot to pass, this causes VM image save and new VM allocation with possible queues after provisioning itself is completed. Possible retry can occur in following cases:

• Agent watchdog expiry (Agent or VM was lost)

• AI determines the failure to be related to infrastructure.

Build

Possible retry can occur in following cases:

• Agent watchdog expiry (Agent or VM was lost)

• VM reset is requested by instructions.

• AI determines the failure to be related to infrastructure.

Test

Possible retry can occur in following cases:

• Agent watchdog expiry (Agent or VM was lost)

• VM reset is requested by instructions.

General retry cases internal to Coin

Some actions will be retried by Coin:

• Uploader will attempt uploads indefinetly, only in case of local file producing file not found error, upload will not be retried for that file.

• Agent’s logger will attempt log push indefinetly. Failure in agent logger(connection to Coin) can quickly also result in the log buffer(1024 rows) to fill, which subsequently blocks agent command execution due to I/O block on process pipe.

• VM allocation is attempted 7 times with 15min timeout for OS boot and 4h timeout waiting for HW.

• Coin<->agent transfers retry 20 times with incremental delay of 5s-2min.

• Windows signing is retried 10 times with incremental delay of 5s-1min.

Design Goals

• reproducible

  • VMs/agents can be re-created from scratch with a single command

  • each build/test run clones a template VM and destroys it afterwards

  • changes to a machine template must pass CI tests in order to be incorporated

• reliable

  • Agents have a way of measuring their health (disk/cpu/ram)

  • notification on missing/lost agents/VMs

  • measuring of build/test run times and comparing them, alarm when they increase a lot

• fast

  • artifacts are used to track the baseline for each repository

  • when a build completes the tests can be run in parallel by several agents

• understandable

  • try to keep things simple

  • no layers of scripts on top of each other

  • non-trivial things require documentation so that it will be easy to make modifications in the future

  • every developer should be able to make a change without getting scared (at least not too much)

• flexible

  • adding a new branch in an existing module (qtbase gets the new 6.1 branch) must work and be tested without change to the buildbot setup

  • adding a new module should require at most one line, ideally it also requires no configuration

  • it is trivial to mark tests or individual test functions as “ignored” and there is still a good indication of what failed but was ignored/insignificant

• helpful

  • early feedback - push a change to gerrit and get instant replies when it breaks compilation on any platform

  • the buildbot will be publicly visible

  • it will be possible to submit try runs of a patch for a selected platform

• informative

  • produces output that helps understanding where we have problems and which tests fail

Setting up the entire infra-structure on a new machine should be straight forward.