Troubleshooting

How can I disable an attribute?

If your Nix expression is used for other purposes than CI, you may consider using a ci.nix to 're-export' only the derivations you care about. For example

ci.nix
let default = import ./default.nix {};
in {
  inherit (default) myPackage1 myPackage2;
}

Alternatively, you can exclude attributes by putting them inside an attribute set without calling pkgs.recurseIntoAttrs. This way, you can still use the attribute for other purposes, but it will not show up in CI. For example:

default.nix
let pkgs = import ./nix {};
in {
  inherit (pkgs) myPackage1 myPackage2;
  shells = {
    # mkShell is designed to always fail its build
    projectShell = pkgs.mkShell {
      buildInputs = [ pkgs.nixops ];
    };
  };
}
shell.nix
(import ./default.nix).shells.projectShell
$ nix-build
/nix/store/...-hello-2.10
$ nix-shell --run 'nixops --version'
NixOps 1.6.1

Why doesn’t my attribute show up in the evaluation?

Typically a derivation may be missing from the traversal when:

  • in a nested attribute set without pkgs.recurseIntoAttrs. This wrapping must be performed at every attribute set; it is not recursive by nature.

  • in a list instead of attribute set. Lists at the top-level are currently not supported. We recommend against this use of lists, because they must be ignored when in an attribute set and because they are often less useful than attributes.

  • in a function in an attribute set. Only the top-level expression will be "auto-called". Functions in attributes are ignored by nix-instantiate. This prevents the needless traversal of library functions.

  • in default.nix while ci.nix or nix/ci.nix is also present, overriding default.nix.

The agent is designed to produce the same output as nix-build/nix-instantiate, without specifying -A.

See How do I build my project locally?.

How do I build my project locally?

Evaluation on Hercules CI follows nix-build quite closely. A good approximation is:

NIX_PATH="" nix-build nix/ci.nix    # or ci.nix or default.nix

Some differences do exist.

You may also want to use the repl to inspect your expressions.

$ NIX_PATH="" nix repl
nix-repl> ci = import ./nix/ci.nix

nix-repl> ci
«lambda @...»

nix-repl> ci = import ./nix/ci.nix {}    # the agent will "auto-call" a function at the root

nix-repl> ci.<TAB>
ci.devTools    # attributes from your ci.nix
ci.frontend

nix-repl> ci.frontend
«derivation /nix/store/n3zg734dldkfpsmrsjrfhympglygv4ge-frontend.drv»

nix-repl> ci.devTools.<TAB>
ci.devTools.arion
ci.devTools.jq
ci.devTools.recurseForDerivations    # This is important

nix-repl> ci.devTools.recurseForDerivations
true    # This means arion and jq will also be found

How do I resolve file 'nixpkgs' was not found in the Nix search path?

Hercules CI currently requires that projects pin their dependencies. The Nix search path, also known as <…​> syntax, NIX_PATH and -I, is empty.

See Pinning in the Repository Setup Guide.

How can I manually trigger a build?

Individual derivations can be restarted from the dashboard.

Evaluation can currently only be restarted by pushing to a branch.

My commit status is stuck in Pending - Waiting for agent

Make sure you have at least one x86_64-linux agent up and running.

If the agent didn’t start, it should leave a message in the system journal. To see the last 100 lines of the journal with NixOps, run nixops ssh agent journalctl -u hercules-ci-agent -n 100

Note that an agent can currently only be assigned to a single account. Assigning agents to multiple accounts is on the roadmap, but for now you will have to run multiple agents to build for multiple accounts.

What are the nix-store --realise processes for?

The agent talks to Nix via nix-store to build derivations. Each of these processes is for building a single derivation, which may include fetching the closure of dependencies.

My derivation is taking long and there’s no build log yet. What’s going on?

Derivation logs are currently only available after the derivation has succeeded or failed.

You may run nix-store --realise /nix/store/<…​>.drv on an agent for troubleshooting purposes.

Why is my derivation status Failure, while the build log looks successful?

Currently some technical errors and configuration errors are not reported all the way to the dashboard. Usually these errors are due to a small mistake in the agent configuration.

We are addressing these issues. In the meanwhile, check the log on your agents: journalctl -u hercules-ci-agent, macOS: /var/log/hercules-ci-agent.log.

Why do some build logs appear in the agent log (journalctl etc) but not all of them?

Only evaluation currently logs to the agent log. Evaluation may include some building if you use import from derivation.

What is the error Exception: path '/nix/store/[…​]' does not exist and cannot be created; type: nix::Error

This may be caused by a misconfiguration.

Try

journalctl -u nix-daemon.service -n 10 -g hercules-ci-agent # where hercules-ci-agent is your agent's system user

It should print something like accepted connection from pid 31134, user hercules-ci-agent (trusted). If (trusted) is missing, your daemon isn’t configured correctly. Follow the steps in My agent logs warning: ignoring untrusted substituter '[…​]'.

My agent logs warning: ignoring untrusted substituter '[…​]'

This means your daemon isn’t configured correctly.

On NixOS, using the supplied NixOS module for the agent:

  • Make sure nix.trustedUsers is not set with lib.mkForce or similar, anywhere in your NixOS configuration or modules.

  • DO NOT set trusted-users via nix.extraOptions.

On other systems, make sure that:

  • /etc/nix/nix.conf has exactly one line starting with trusted-users =

  • /etc/nix/nix.conf has hercules-ci-agent (or your agent’s system user) in trusted-users =, e.g. trusted-users = root @wheel hercules-ci-agent

An agent running on macOS reports exception:Network.Socket.getAddrInfo but DNS works fine

This can be a symptom of the default macOS power saving settings. See Configure macOS for build server duty

Unfortunately, macOS has negative DNS caching, which means that a temporary failure in your DNS will be remembered, rendering the agent’s built in retry behavior ineffective.

It [appears](discussions.apple.com/thread/252332579) that this behavior can not be disabled in macOS.

Otherwise, a one-off situation can be resolved by running

sudo dscacheutil -flushcache