Hercules CI Documentation

Hercules CI is a continuous integration service for Nix. In order to provide this service, it connects the following components:

  • hercules-ci.com

  • github.com

  • your agent machines

This document describes the steps to set up the integration. The process should be self-explanatory, guided by the hercules-ci.com web application. If you find yourself stuck, or if you want to know the process in advance, this document is intended to provide reference.

Warning

This is preview software. We’re gathering feedback to improve the service. It is not meant to be relied upon just yet.

Have a look at our roadmap to know what (not) to expect.


Setup Guide

Step 1. GitHub Setup

Sign in

  1. Click the GitHub "Sign in" button on the top right of hercules-ci.com and follow the steps.

  2. Hercules-ci.com will now show the application instead of the regular front page.

Install the GitHub app

Having signed in, you are ready to add Hercules CI to your GitHub user account or to a GitHub organization account.

  1. Click the "Connect GitHub repositories" button near the top of the dashboard.

  2. GitHub will give you a choice to install the application on all repositories, which includes repositories to be created in the future, or to select specific repositories. We recommend that you make a selection of repositories, assuming not all your repositories have suitable Nix expressions. Follow the steps.

  3. Write a default.nix in the root of your repository.

    If default.nix is a function, it must define default arguments in order to be auto-callable. NIX_PATH will be empty.

    Hercules CI behaves similarly to nix-build. default.nix can provide multiple derivations by returning an attribute set. Nested attribute sets can be traversed by calling Nixpkgs' pkgs.recurseIntoAttrs.

  4. Test it locally by running the command

    myrepo/$ NIX_PATH= nix-build --no-out-link
  5. Commit and push, so Hercules CI will pick it up.

Collaborators

Hercules CI will automatically pick up permissions from GitHub. Collaborators will find the repositories they have access to when they sign in.

Step 2. Agent Setup

The agent adds itself to a cluster by means of a cluster join token. The same token may be used to add multiple agents.

Warning

When updating the agent token on a previously deployed agent, you will need to remove the session.key file and restart the agent in order to start using the new cluster join token. This file is stored in /var/lib/hercules-ci-agent/secretState/session.key by default. This will be resolved in issue #8

Choose one of the deployment options:

  • NixOps, for existing or provisioning NixOS machines

  • NixOS, for existing NixOS machines or custom deployment tools

  • Nix-darwin, the recommended configuration management tool for Mac OS X

  • Manually, when the other deployment methods are not suitable

Deploy with NixOps

1. Create a folder called mycompany-agents with a file:

# hercules-ci-agents.nix
let
  hercules-ci-agent =
      builtins.fetchTarball "https://github.com/hercules-ci/hercules-ci-agent/archive/stable.tar.gz";
in
{
  network.description = "Hercules CI agents";

  agent = {
    imports = [
      (hercules-ci-agent + "/module.nix")
    ];

    services.hercules-ci-agent.enable = true;
    services.hercules-ci-agent.concurrentTasks = 4; # Number of jobs to run
    services.hercules-ci-agent.binaryCachesFile = ./binary-caches.json;
    deployment.keys."cluster-join-token.key".keyFile = ./cluster-join-token.key;
  };
}

2. Choose between backends. NixOps can deploy to:

An example physical specification:

# hercules-ci-agents-target.nix
{
  agent = {
    deployment.targetHost = "10.0.0.42"; # Your agent's IP address running NixOS
  };
}

3. Get a cluster join token.

  • In the dashboard, find the account for which you would like to deploy the agent,

  • Click the "Agents" button and the button in "Generate token" tab. This produces a private token that should be protected like a password.

  • Copy the token into a plain text file cluster-join-token.key in the same directory. Protect this file like a password.

3. Write a binary caches file, binary-caches.json in the same directory as hercules-ci-agents.nix.

4. Deploy using:

$ nix-shell -p nixops -I nixpkgs=http://nixos.org/channels/nixos-19.03/nixexprs.tar.xz
$ nixops create -d my-agent ./hercules-ci-agents.nix ./hercules-ci-agents-target.nix
$ nixops deploy -d my-agent

The agent will start working as soon as the deployment has succeeded and builds are queued.

Troubleshooting

To inspect the agent’s local log, run nixops ssh agent journalctl -u hercules-ci-agent -n 100 to see the last 100 lines.

Deploy with NixOS

NixOS deployment can be performed on the target machine itself or remotely by means of nixos-rebuild --target-host.

These instructions will use /etc/nixos as the location for the configuration modules. This directory is only used at evaluation time. If you perform remote deployment, you will use some project directory in place of /etc/nixos.

1.

Add this configuration to /etc/nixos/configuration.nix:

let
  hercules-ci-agent =
      builtins.fetchTarball "https://github.com/hercules-ci/hercules-ci-agent/archive/stable.tar.gz";
in
{
  imports = [
    (hercules-ci-agent + "/module.nix")
  ];

  services.hercules-ci-agent.enable = true;
  services.hercules-ci-agent.concurrentTasks = 4; # Number of jobs to run
}

2. Get a cluster join token.

  • In the dashboard, find the account for which you would like to deploy the agent,

  • Click the "Agents" button and the button in "Generate token" tab. This produces a private token that should be protected like a password.

  • Copy the token into a plain text file /var/lib/hercules-ci-agent/secrets/cluster-join-token.key on the target machine.

3. Write a binary caches file, binary-caches.json. Continue with an applicable section below.

Local deployment

If you’re deploying to the local machine, move it to /var/lib/hercules-ci-agent/secrets/binary-caches.json and add this line to your /etc/nixos/configuration.nix:

services.hercules-ci-agent.binaryCachesFile = /var/lib/hercules-ci-agent/secrets/binary-caches.json;

Make sure that /var/lib/hercules-ci-agent/secrets can only be read by hercules-ci-agent.

Remote deployment

If you’re deploying to a remote machine, you need to store it in two locations. One for evaluation, in the directory of your configuration.nix file and one for the deployed agent to read. Add this line to your configuration.nix:

services.hercules-ci-agent.binaryCachesFile = ./binary-caches.json;

Install the same file on the target machine as /var/lib/hercules-ci-agent/secrets/binary-caches.json.

Make sure that /var/lib/hercules-ci-agent/secrets can only be read hercules-ci-agent.

Tip

If your deployment solution lacks a method to store configuration files confidentially, you may choose to maintain the binary-caches.json file separately in both locations. This allows you to censor the secrets like {"signingKeys": [], "authToken": ""} in the file used for evaluation only.

Make sure not to lose the signing keys and do deploy them to the target machine!

4. Apply the configuration by running nixos-rebuild switch.

Troubleshooting

To inspect the agent’s local log, run journalctl -u hercules-ci-agent -n 100 on the target machine to see the last 100 lines.

Deploy manually with a configuration file

If the other deployment methods do not suit your needs, you may choose to run or deploy the agent manually.

1. Send us an e-mail at [email protected]. We can give advice and we are interested in expanding our officially supported deployment methods.

2. Write an agent.toml file. Most of the entries are optional. A small example:

baseDirectory = "/var/lib/hercules-ci-agent"
concurrentTasks = 4

This guide assumes that the baseDirectory is set to /var/lib/hercules-ci-agent.

3. Get a cluster join token.

  • In the dashboard, find the account for which you would like to deploy the agent,

  • Click the "Agents" button and the button in "Generate token" tab. This produces a private token that should be protected like a password.

  • Copy the token into a plain text file /var/lib/hercules-ci-agent/secrets/cluster-join-token.key.

4. Write a binary caches file and install it in /var/lib/hercules-ci-agent/secrets/binary-caches.json.

5. Configure the system to use the caches and accept their public keys. When using a private cache, you will also need to install an appropriate netrc file in /etc/nix/daemon-netrc.

6. Start the agent with the command hercules-agent --config agent.toml, preferably via some process supervision system.


FAQ / How To

My commit status is stuck in Pending - Waiting for agent

Make sure you’ve deployed an agent. If the agent didn’t start, it should leave a message in the system journal. To see the last 100 lines of the journal with NixOps, run nixops ssh agent journalctl -u hercules-ci-agent -n 100

Note that an agent can currently only be assigned to a single account. Assigning agents to multiple accounts is on the roadmap, but for now you will have to run multiple agents to build for multiple accounts.

Can I disable an attribute? I don’t want to build it on CI.

You can exclude attributes by putting them inside an attribute set without recurseIntoAttrs. This way, you can still use the attribute for other purposes, but it will not show up in CI. For example:

# default.nix
let pkgs = import ./nix {};
in {
  inherit (pkgs) myProject;
  shells = {
    projectShell = pkgs.mkShell {
      buildInputs = [ pkgs.nixops ];
    };
  };
}
# shell.nix
(import ./default.nix).shells.projectShell
$ nix-build
/nix/store/...-hello-2.10
$ nix-shell --run 'nixops --version'
NixOps 1.6.1

My attribute doesn’t show up in the evaluation

Is it in a nested attribute set? Make sure you call pkgs.recurseIntoAttrs on all levels of nested attrsets.

Does nix-build build the attribute with specifying -A? If it doesn’t, Hercules CI shouldn’t build it either. Otherwise, please contact us.

How can I manually trigger a build?

You can trigger a rebuild of a derivation in the web interface. Evaluations can be triggered by pushing a branch. You may use a temporary branch just to trigger an evaluation and commit status update.

Where is the evaluation log?

A log of the evaluation is not available yet. Normally this should not be needed, however, we do want to show the builtins.trace messages in the future.

To troubleshoot unexpected technical failures of evaluation on the agent, you may inspect the agent log.

My derivation is taking long and there’s no build log yet. What’s going on?

Derivation logs are currently only available after the derivation has succeeded or failed.

You may run nix-store --realise /nix/store/<…​>.drv on an agent for troubleshooting purposes.

What are the nix-store --realise processes about?

The agent talks to Nix via nix-store to build derivations. Each of these processes is for building a single derivation, which may include fetching the closure of dependencies.

Some build logs appear in the agent log (journalctl etc) but not all of them?

Only evaluation currently logs to the agent log. Evaluation may include some building if you use import from derivation.


Reference

Evaluation

Evaluation is the first step when it comes to building your project. It is performed by your agent as follows:

  1. At the top-level of your repository, the agent will search for a Nix file to evaluate in the following order: nix/ci.nix, ci.nix or default.nix

  2. NIX_PATH will be empty. Make sure you pin nixpkgs and not rely on <nixpkgs> search path. This is to ensure reproducability and purity also when it comes to nixpkgs

  3. If any of the attributes fail to evaluate, job evaluation phase will be marked as failed and error messages will be showed inline per attribute

Agent configuration file

When deploying with NixOS, NixOps or nix-darwin you should use the module documentation instead, or skip ahead to read about less frequently used options to use with extraOptions.

The configuration format for the agent is TOML markup. Its location must always be specified by invoking hercules-ci-agent --config agent.toml.

A basic agent.toml looks as follows:

agent.toml

baseDirectory = "/var/lib/hercules-ci-agent"
concurrentTasks = 4

apiBaseUrl

Optional. Defaults to https://hercules-ci.com.

HTTP API agent will connect to.

baseDirectory

Required.

Directory with all the agent state: secrets, work, etc.

binaryCachesPath

Optional. Defaults to staticSecretsDirectory/binary-caches.json if that file exists.

Path to a JSON file containing binary cache secret keys.

Currently only supports Cachix binary caches.

Note that binary cache configuration affects the whole system’s Nix configuration.

Make sure that your system accepts the binary cache. We currently recommend to look at the sources for details.

Example:

binary-caches.json

{ "my-private":
    { "kind": "CachixCache",
      "authToken": "eyJhaf23GH53a.bc23BUSI.9q3048hWHh"
    , "publicKeys": ["my-private.cachix.org-1:EjBSHzF6VmDnzqlldGXbi0RM3HdjfTU3yDRi9Pd0jTY="]
    , "signingKeys": ["XXX"]
    }
}

clusterJoinTokenPath

Optional. Defaults to staticSecretsDirectory/cluster-join-token.key.

Path to a secret token retrieved when creating a new agent via https://hercules-ci.com/dashboard.

This token is used for authenticating to apiBaseUrl.

concurrentTasks

Optional. Defaults to 4.

Combined number of workers to use for building and evaluating Nix derivations.

The optimal value depends on the resource consumption characteristics of your workload, including memory usage and in-task parallelism. This is typically determined empirically.

staticSecretsDirectory

Optional. Defaults to baseDirectory/secrets.

This is the default directory to look for statically configured secrets like clusterJoinTokenPath.

workDirectory

Optional. Defaults to baseDirectory/work.

The directory in which temporary subdirectories are created for task state. This includes sources for Nix evaluation.