Overview

Structure

The Qt Continuous Integration project (CI) is a collection of scripts that work together to provide a complete continuous integration environment for projects with complex dependencies.

At the heart of the CI is the scheduler. It receives events (such as a commit made available in gerrit) and processes them. Dependencies are deduced by the dependency resolver which is queried by the scheduler in order to build a set of work items. Let’s assume module B should be tested and it depends on module A. The dependency resolver will maintain git checkouts of all modules. It then looks at the ‘sync.profile’ in the root directory of B and from there sees that A is required as dependency. The scheduler then uses this information to schedule a build of A, a build of B which waits for A to complete, and a test run for B.

The system is build with artifacts in mind, it doesn’t just build a module over and over. Instead, when we ask to test module B again, it will see that there are already build artifacts for module A. Unless A changed of course. But assuming the system is used to control the repository of A as well, it will always have an up to date build of A. The storage module is responsible for managing the artifacts and build logs.

digraph ci_overview { "gerrit monitor" -> "scheduler"; "scheduler" -> "dependency resolver"; "dependency resolver" -> "repository manager"; "repository manager" -> "dependency resolver"; "dependency resolver" -> "scheduler"; "scheduler" -> "agent pool"; "agent pool" -> "agent 1"; "agent pool" -> "agent 2"; "agent pool" -> "agent 3"; "agent 1" -> "scheduler"; "agent 2" -> "scheduler"; "agent 3" -> "scheduler"; "scheduler" -> "integrator"; "scheduler" -> "storage"; }

At this point in time the system is tightly coupled to gerrit and git as source control systems.

The scripts discover each other using a simple RPC name service, which is bootstrapped using an environment variable or a command line parameter.

The scripts communicate with each other using Apache Thrift over TCP sockets as RPC mechanism.

Interaction with Gerrit

Coin interacts with gerrit in two ways. Firstly the changes are polled periodically and Coin will create integrations from staged commits. Then the changes move to integrating state, which tells that coin is working on them. After that commits are either merged or rejected and returned to new state.

Coin also listens to gerrit event stream, this is where coin will pick up the precheck events. The default precheck configurations are listed in the precheck.yaml in each product repo, in most cases the product is qt5. See platform configurations for details.

Precheck can also define the platforms as a list and Coin will parse the platforms based on that.

digraph gerrit_overview { rankdir=LR; node [shape=diamond]; "timeout"; node [shape=box]; "gerrit-stream-events" -> gerrit_monitor; gerrit_monitor -> gerrit_monitor [label="process_event"]; gerrit_monitor -> timeout; timeout -> "gerrit-staging-new-build"; timeout -> scheduler; scheduler -> scheduler [label="test"]; scheduler -> integrator; integrator -> "gerrit-staging-approve"; }

Debugging

To get more verbose debug output, use the –verbose option for each script. Alternatively you can export an environment variable before running any of the scripts:

export QT_CI_DEBUG=*

to only increase debug output in one area:

export QT_CI_DEBUG=CI.Scheduler

or several areas:

export QT_CI_DEBUG="CI.Scheduler CI.Application CI.Storage"

Vocabulary

Term

Definition

CI

Continuous Integration

agent

A build slave. Identified by host name and port.

project

namespace/project_name: A git module that is built and tested with the CI. For example each of Qt’s git modules is a project. A project must always include its namespace. Examples are “qt/qtbase”, “qt/qtdeclarative”, it is always a string.

change

gerrit_instance, project, ref, branch: A change is a patch in a project. It is defined by project and sha1. To be useful in the context of the CI, each change has a branch associated. The branch is the target when testing a change. For testing each change will be cherry-picked on top of it’s target branch and if found good, it is cherry-picked into the project’s git repository. In addition each change keeps a reference to which gerrit instance it belongs.

buildKey

project/sha1/test_configuration_key/dependency_hash/request_type: A string that represents the build or testing of a change. It encodes the change including dependencies (a hash of all dependencies’ sha1s).

baseKey

project/sha1/test_configuration_key/dependency_hash: This is provided for convenience and represents the part of ‘buildKey’ that is shared between different ‘WorkItem’ types with identical dependency paths.

WorkItem

A task internal to the scheduler. It is either a build, test or final item. Build items represent the building of one change, test items running the tests for a change and final items collect results from all items needed to approve or disapprove a change.

Design Goals

  • reproducible

    • build slaves can be re-created from scratch with a single command

    • each build/test run clones a template VM and destroys it afterwards

    • changes to a machine template must pass CI tests in order to be incorporated

  • reliable

    • slaves have a way of measuring their health (disk/cpu/ram)

    • notification on missing slaves

    • measuring of build/test run times and comparing them, alarm when they increase a lot

  • fast

    • artifacts are used to track the baseline for each repository

    • when a build completes the tests can be run in parallel by several slaves

  • understandable

    • try to keep things simple

    • no layers of scripts on top of each other

    • non-trivial things require documentation so that it will be easy to make modifications in the future

    • every developer should be able to make a change without getting scared (at least not too much)

  • flexible

    • adding a new branch in an existing module (qtbase gets the new 6.1 branch) must work and be tested without change to the buildbot setup

    • adding a new module should require at most one line, ideally it also requires no configuration

    • it is trivial to mark tests or individual test functions as “ignored” and there is still a good indication of what failed but was ignored/insignificant

  • helpful

    • early feedback - push a change to gerrit and get instant replies when it breaks compilation on any platform

    • the buildbot will be publicly visible

    • it will be possible to submit try runs of a patch for a selected platform

  • informative

    • produces output that helps understanding where we have problems and which tests fail

Setting up the entire infra-structure on a new machine should be straight forward.