.. _configuration: Coin Configuration ================== Qt CI for module maintainers ---------------------------- This section contains instructions for configuring a module to be built in the Coin CI. The module configuration consists of dependencies and instructions in YAML files that both live in the module repository and will be automatically recognized by Coin. The module dependencies must declare all dependencies that a module needs to be compiled and have its tests run. .. note:: Coin uses pyyaml parser which conforms yaml 1.1 spec. YAML module dependencies ~~~~~~~~~~~~~~~~~~~~~~~~ A module's dependencies can be defined in a YAML file, in the top-level directory of the repository. Example 1. Module with two dependencies. repo.git:dependencies.yaml:: dependencies: ../qtbase.git: ref: 8155d0693f596c6b73d0acd977d5077dd7a6e6ea required: true ../qtsvg.git: ref: 332f9d686a08948bde82c925701a48150f29c10d required: false Example 2. If the module has no dependencies, the dependencies are an empty dictionary. dependencies.yaml:: dependencies: {} Definitions:: dependencies: dict Defines a dictionary of git repositories. key: string URL (relative or absolute) of dependency's git repository. ref: string Git reference of the dependency's git repository (sha1). required: bool If set to true, the repository is direct dependency, which means that the module will not compile without it. If set to false, the dependency is not strictly required, but will be installed during build. Downstream check ~~~~~~~~~~~~~~~~ Downstream check will test downstream modules with current module's sha1 updated as dependency. If the check fails, a message is posted in gerrit to the upstream module change. Example:: downstream_check: mode: build # build or test modules: # Modules listed as in dependencies.yaml, dependencies in the chain will be added automatically. ../qtdeclarative: ref: dev configurations: # List of configuration IDs to limit the scope, omit for all configs. - RHEL-8.4-host Git module dependencies (deprecated) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ The old module dependency declaration was defined in .gitmodules file that is internally used by git. Example. Repository qt/qt5.git git modules file. qt5.git:.gitmodules:: ... [submodule "qtbase"] path = qtbase url = ../qtbase.git branch = dev status = essential [submodule "qtsvg"] depends = qtbase path = qtsvg url = ../qtsvg.git branch = dev status = addon [submodule "qtdeclarative"] depends = qtbase recommends = qtsvg path = qtdeclarative url = ../qtdeclarative.git branch = dev status = essential ... Module configuration #################### Module configuration is a YAML file which contains instructions that will be executed on the cloned VM. The file specifies accepted platforms, build instructions and test instructions. The relevant module configuration is all in one place, in coin folder of the module git repository: repo.git: coin/module_config.yaml Alias ~~~~~ Alias is used to change the name of the module. This affects the exported binary and sources package names. In modules with dependencies.yaml and alias set in qt5 .gitmodules, the alias should be also set in the module. This allows reusing the artifacts from module integrations in qt5 integrations. Alias is added on the root level:: version: 2 alias: qtbase Includes ######## It is possible to include configurations from another modules or from modules own shared folder. module can have shared instructions under coin/instructions folder. Include command can be used at any part of module configuration where it is wanted to be included. These can be referenced with:: # Qtbase could have instructions under coin/instructions/prepare_build_env.yaml. # This can be then referenced with - !include "{{qt/qtbase}}/prepare_build_env.yaml" There are some limitations to includes: - Anchors are not working across includes. - The included file needs to be defined in the repository or one of it's dependencies, in particular product lookup is not possible. Platforms ######### accept_configuration tells what platforms are accepted, it has only one condition which can be nested. Typical module_config file consisting many and conditions under one or condition to specify platforms:: version: 2 accept_configuration: condition: or conditions: - condition: and conditions: - condition: property property: host.osVersion in_values: [MacOS_10_13, RHEL_7_4] - condition: property property: host.osVersion equals_property: target.osVersion - condition: and conditions: ... instructions: Build: - type: SetBuildDirectory directory: "{{.SourceDir}}" - ... Test: - type: EnvironmentVariable variableName: Dummy variableValue: dummy - ... Machine Type ############ Machine size can be defined in yaml files in coin/platform_configs directory. COIN allows one to add features for configurations, which modifies the default VM resources. At the momenent there are four different features defined: - "VMSize4" - "VMSize8" - "VMSize16" - "DoubleSizeVM" First three sets explicitly the number of cores allocated for the machine. Fourth one finally doubles the amount after all other rules are handled. The machine configuration in platform confiiguration yaml affects to all modules executing that configuration. Features can be added to platform configurations like this:: Configurations: - Template: 'qtci-linux-Ubuntu-20.04-x86_64-50' Compiler: 'GCC' Features: ['Sccache', 'VMSize8'] Configure arguments:.... When adding extra resources for the configuration, one should keep in mind that even if the one target might get executed faster, the whole integration may take even longer. This is because each host has limited amount of resources and even with moderate load, the requested VM may take longer to find host with enough space to fit in. Machine size can also be defined in the module_config.yaml with machine_type property. This affects only to module itself. machine_type field has two entries 'Build' and 'Test' which define core count for build and test phases. Amount of RAM is calculated off from the core count with formula '3GB + core_count * 3GB'. Machine type spec allows defining only one of the entries, with the other one defaulting to either predetermined values in Coin or ultimately to 2 cores if no special rules are defined. Typical machine_type section looks like:: version: 2 accept_configuration: ... machine_type: Build: cores: 8 Test: cores: 2 Tags #### Tags are used to adjust some of the CI behavior during module builds and tests. +-----------------+-------------------------------------------------------------------------------------------------------------+ | Tag | Options | +=================+=============================================================================================================+ | Git | Instead of extracting source archive on the VM, execute a git clone. This enables source control on the VM. | +-----------------+-------------------------------------------------------------------------------------------------------------+ | Documentation | If set adds top level documentation workitem as dependency to configurations with documentation feature | +-----------------+-------------------------------------------------------------------------------------------------------------+ Instructions ##################### Instruction types ~~~~~~~~~~~~~~~~~ Each instruction must have type, required options and optionally common options. +----------------------------------+----------------------------------------+--------------------------------+ | Instruction type | Options | Type | +==================================+========================================+================================+ | ExecuteCommand | command | List | | +----------------------------------------+--------------------------------+ | | userMessageOnFailure | String | | +----------------------------------------+--------------------------------+ | | executeCommandArgumentSplitingBehavior | ExecuteCommand_argument_split_ | | +----------------------------------------+--------------------------------+ | | resetOnFailure_ | Optional | +----------------------------------+----------------------------------------+--------------------------------+ | | ChangeDirectory | directory | String | | | MakeDirectory | | | | | SetBuildDirectory | | | +----------------------------------+----------------------------------------+--------------------------------+ | WriteFile | filename | String | | +----------------------------------------+--------------------------------+ | | fileContents | String | | +----------------------------------------+--------------------------------+ | | hiddenContent | Bool | | +----------------------------------------+--------------------------------+ | | fileMode | Int|Octal e.g 0755 | +----------------------------------+----------------------------------------+--------------------------------+ | | EnvironmentVariable | variableName | String | | | AppendToEnvironmentVariable +----------------------------------------+--------------------------------+ | | PrependToEnvironmentVariable | variableValue | String | +----------------------------------+----------------------------------------+--------------------------------+ | | InstallSourceArchive_ | relativeStoragePath | String | | | InstallBinaryArchive_ +----------------------------------------+--------------------------------+ | | InstallTestBinaryArchive | directory | String | +----------------------------------+----------------------------------------+--------------------------------+ | SetExecutionPhaseName | executionPhaseName | String | +----------------------------------+----------------------------------------+--------------------------------+ | RunQtUnitTest | directory | String | | +----------------------------------------+--------------------------------+ | | runTestCommand | List | | +----------------------------------------+--------------------------------+ | | testRepetitionAllowance | Int | +----------------------------------+----------------------------------------+--------------------------------+ | | UploadArtifact_ | archiveDirectory | String | | | UploadTestArtifact +----------------------------------------+--------------------------------+ | | transferType | transferType | +----------------------------------+----------------------------------------+--------------------------------+ | UploadTestPlan | generalTestPlanPath | String | | +----------------------------------------+--------------------------------+ | | qtestlibTestPlanPath | String | +----------------------------------+----------------------------------------+--------------------------------+ | PrepareConanBuildInfos | sourceDirectory | String | +----------------------------------+----------------------------------------+--------------------------------+ | SignPackage | directory | String | +----------------------------------+----------------------------------------+--------------------------------+ | Rename | sourcePath | String | | +----------------------------------------+--------------------------------+ | | targetPath | String | +----------------------------------+----------------------------------------+--------------------------------+ | ScheduleUploadTestResults | | - | | +----------------------------------+----------------------------------------+--------------------------------+ | ExecuteTestPlan | runTestCommand | String | | +----------------------------------------+--------------------------------+ | | relativeStoragePath | String | +----------------------------------+----------------------------------------+--------------------------------+ | Other | instructionName | String | +----------------------------------+----------------------------------------+--------------------------------+ .. _ExecuteCommand_argument_split: ExecuteCommand argument split ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ExecuteCommand can split given arguments in two ways, before or after variables have been substituted. By default split happens before subsitution if a string is supplied and in the case of a array it is kept as it is. For example:: Env.arg = "foo bar" # By default if a string is supplied to executeCommand it is split before substitution. ["-I {{.Env.arg}}"] ["-I", "foo bar"] # If a list is supplied to ExecuteCommand it is kept as it is and substituted. ["-I", "-j1 {{.Env.arg}}"] ["-I", "-j1 foo bar"] # One can also opt in for splitting after subsitution. ["-I {{.Env.arg}}"] ["-I", "foo", "bar"] # If a list is supplied with split after option each item will be still split individually. ["-I", "-j1 {{.Env.arg}}"] ["-I", "-j1", "foo", "bar"] Split behavior is defined with option executeCommandArgumentSplitingBehavior, default behavior being SplitBeforeVariableSubstitution. +--------------------------------------------+----------------------------------------+ | Option | Values | +============================================+========================================+ | executeCommandArgumentSplitingBehavior | SplitBeforeVariableSubstitution | | | SplitAfterVariableSubstitution | +--------------------------------------------+----------------------------------------+ .. _resetOnFailure: Reset on failure ~~~~~~~~~~~~~~~~ Reset on failure will change how to handle command execution failure. If set to true and the command fails, the VM will be deleted and the job restarted on new VM. Common options for instructions ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Instructions can have common fields, namingly timeouts, error message and enable condition. Timeouts are required to be set only for long running commands, since they default to 10 seconds. UserMessageOnFailure should be filled in. It is not always required (some of the commands have default message) but it is highly recommended for debugging purposes. The message should contain information about: - which step was attempted (for example: could not create directory "foo") - what was the reason for the step (for example: the foo directory is need for shadow builds) - how a user can resolve the probem (for example: There is not known reasons for this operation to fail, please contact CI admins) - contain information if the issue is most likely caused by code or by infrastructure (for example: creating a directory failure is likely infra problem while failure in tests execution not) +-----------------------+-----------+---------+---------------------------------------------------------------------+ | Field | Type | Default | Description | +=======================+===========+=========+=====================================================================+ | maxTimeInSeconds | Int | 10 | Maximum command runtime in seconds before timeout. | +-----------------------+-----------+---------+---------------------------------------------------------------------+ | maxTimeBetweenOutput | Int | 10 | Maximum time in seconds between output from command before timeout. | +-----------------------+-----------+---------+---------------------------------------------------------------------+ | userMessageOnFailure | String | Varies | Reported error message if command fails. | +-----------------------+-----------+---------+---------------------------------------------------------------------+ | enable_if | Condition | None | If provided tells whenever to include instruction. | +-----------------------+-----------+---------+---------------------------------------------------------------------+ | disable_if | Condition | None | If provided tells whenever to exclude instruction. | +-----------------------+-----------+---------+---------------------------------------------------------------------+ | executeOn | String | success | See executeOndetails_ | +-----------------------+-----------+---------+---------------------------------------------------------------------+ | executeOnTarget | String | success | See executeOndetails_ | +-----------------------+-----------+---------+---------------------------------------------------------------------+ | resetError | String | None | See resetError_ | +-----------------------+-----------+---------+---------------------------------------------------------------------+ .. _executeOndetails: ExecuteOn details ~~~~~~~~~~~~~~~~~ ExecuteOn controls when the instruction is run in relation to success or failure of previous instructions. The table below shows the inputs and their actions. +------------+----------------------------------------------------------------------+ | Value | Action | +============+======================================================================+ | failure | Run only when any of the previous instructions fail. | +------------+----------------------------------------------------------------------+ | success | (default) Run only when all previous instructions pass. | +------------+----------------------------------------------------------------------+ | always | Run always, regardless of pass/fail of previous instructions. | +------------+----------------------------------------------------------------------+ Additionally one can target some instruction to limit the scope by using the instructionName as value to executeOnTarget:: instructionName: "Copy logs" instructionName: "Copy logs fallback" executeOn: "failure" executeOnTarget: "Copy logs" .. _resetError: Reset error ~~~~~~~~~~~ This allows instructions to reset previous failures that happened in the same run. Most of the time it is paired with executeOnTarget to target some particular command and provide fallback etc. usage:: resetError: target +------------+----------------------------------------------------------------------+ | Value | Action | +============+======================================================================+ | target | Only reset failure of instruction targeted by executeOnTarget | +------------+----------------------------------------------------------------------+ | all | Reset all errors that happened in the run before this instruction | +------------+----------------------------------------------------------------------+ .. _InstallSourceArchive: InstallSourceArchive ~~~~~~~~~~~~~~~~~~~~ The install source archive command can take up to 5 inputs. Project, ref and directory are required where gitRemote and excludeSubModules are optional. Gitremote tells the location of the repo, if not defined gitremote defaults to same as in the integration. There are few keywords for the ref and project. CURRENT_BRANCH in ref field will refer to current branch. Project can be PRODUCT, this will result in qt/qt5 or qt/tqtc-qt5 in most cases. PRODUCT_REF will take the sha1 from the product in the integration, this effectively locks the ref for the integration:: type: InstallSourceArchive gitRemote: qt-project project: PRODUCT ref: PRODUCT_ref directory: qt5-repo maxTimeInSeconds: 300 maxTimeBetweenOutput: 300 excludeSubModules: true userMessageOnFailure: "Could not install x source archive." .. _InstallBinaryArchive: InstallBinaryArchive ~~~~~~~~~~~~~~~~~~~~ The install binary archive command takes directory and either filename or regex input. The artifacts that can be installed with the command are all artifacts in the dependency chain from previous builds. The directory specifies where to install the archives. Filename is used to install a single archive. Regex can be used instead of filename if multiple archives are wanted in same directory. If directory is not absolute the directory is prefixed with ~/work:: type: InstallBinaryArchive directory: doc_tools filename: doc_tools.tar.zst maxTimeInSeconds: 300 maxTimeBetweenOutput: 300 For installing e.g. {{.Env.PLATFORM_ID}}_intermediate_files.tar.zst etc:: type: InstallBinaryArchive directory: intermediate_files regex: .*intermediate_files.tar.zst maxTimeInSeconds: 300 maxTimeBetweenOutput: 300 .. _UploadArtifact: UploadArtifact ~~~~~~~~~~~~~~~~~~~~ The upload artifact command can take GenericArtifactWithName as parameter to transferType. This allows custom filename to be used, which can be used to download the artifact in later builds:: type: UploadArtifact archiveDirectory: "{{.BuildDir}}/foobar" transferType: GenericArtifactWithName filename: "{{.Env.COIN_FILENAME}}" maxTimeInSeconds: 1200 maxTimeBetweenOutput: 1200 Instruction groups ~~~~~~~~~~~~~~~~~~ Instruction groups can be created by defining item with type group and instructions field. Groups are useful in sharing enable_if for multiple instructions and yaml anchors for deduplicating instructions between build and test:: type: Group instructions: ... enable_if: ... Group and/or instructions can be tied to anchor per yaml specification:: gcc_make_instructions: &gcc_make_instructions type: Group instructions: ... # Which can be later referenced with - *gcc_make_instructions Conditions ###################### Conditions are used to define accepted platforms and inclusion of instructions. instructions and groups use enable_if to specify their condition. typical instruction with a condition:: - type: SignPackage directory: "package/dir" enable_if: condition: property property: host.os equals_value: Windows Condition types ~~~~~~~~~~~~~~~ Multiple condition types exists. This makes it possible to have nested conditions. Property condition has rule under it and (and, or) conditions have list of conditions specified under them. +----------------+----------------------------------------------------------+ | Condition type | Description | +================+==========================================================+ | property | Condition which specifies rule for certain property. | +----------------+----------------------------------------------------------+ | runtime | Condition which is evaluated at runtime. | +----------------+----------------------------------------------------------+ | and | All child conditions must return true. | +----------------+----------------------------------------------------------+ | or | True when any of child conditions returns true. | +----------------+----------------------------------------------------------+ Condition Properties ~~~~~~~~~~~~~~~~~~~~ Property conditions have a property from workitemconfiguration which then is evaluated with specified rule. Workitemconfiguration and the possible values for platforms and features are specified in :ref:`properties`. Workitemconfiguration fields can be used in property and subproperties of those accessed with a dot. Condition rules ~~~~~~~~~~~~~~~ Property condition can use different rules but only one at a time. These are: +-----------------------+-----------+---------------------------------------------------------------+ | Condition rule | Type | Description | +=======================+===========+===============================================================+ | equals_value | Any | True if given value equals the condition property. | +-----------------------+-----------+---------------------------------------------------------------+ | not_equals_value | Any | True if given value does not equals the condition property. | +-----------------------+-----------+---------------------------------------------------------------+ | equals_property | Any | True if given property equals the condition property. | +-----------------------+-----------+---------------------------------------------------------------+ | not_equals_property | Any | True if given property does not equal the condition property. | +-----------------------+-----------+---------------------------------------------------------------+ | in_values | List | True if condition property matches any value in given list. | +-----------------------+-----------+---------------------------------------------------------------+ | not_in_values | List | True if condition property does not exists in given list. | +-----------------------+-----------+---------------------------------------------------------------+ | contains_value | Any | True if condition property contains given value. | +-----------------------+-----------+---------------------------------------------------------------+ | not_contains_value | Any | True if condition property does not contain given value | +-----------------------+-----------+---------------------------------------------------------------+ Runtime conditions ~~~~~~~~~~~~~~~~~~ Runtime conditions can be used to include or exclude instructions based on environment variables. Only equals_value, not_equals_value, contains_value and not_contains_value operators are accepted for runtime conditions. Example:: enable_if: condition: runtime env_var: foo-bar equals_value: baz Environment Variables ~~~~~~~~~~~~~~~~~~~~~ Environment variables can be referred in multiple ways in coin instructions. The regular environment variables can be referred like:: "{{.Env.}}" Coin also offers some variables related to build environment, which are not in regular environment variables these are agentWorkingDir, sourceDir, buildDir, testDir, installDir, installRoot and testResultsDir. These can be used in two ways either with {{.