Nix: reproducible build

Author: Nick Sigulya

Today we continue our series of posts about Nix and the way we use it at Typeable.

You can find our first post about Nix basics here.

As we like Haskell very much and often use it for development, we’ll use it for our sample application, but you don’t need to have any knowledge of Haskell. After some small fixes, the code shown in the examples can also be used to build projects in other languages.

You can find the complete code for this post in this Github repository.

Problem

Building and CI are some of the biggest problems in software development. Support for build infrastructure often devours an incredible amount of resources. We will use Nix and try to make the situation at least a little more tolerable, even if it’s not possible to amend it. Nix allows us to achieve the reproducibility of builds in our projects, portability between different OSs, unification of components written in different languages, and so on.

Our application

So, let's start with the application we want to build. In our case, it will be a simple program in Haskell displaying the message “Hello, world”.

Our Main.hs:

module Main where

main :: IO ()
main = putStrLn "Hello, World!"

To build a project without Nix, we use stack utility (for more details on the utility see here). The description of our project for stack requires stack.yaml file containing the list of our packages and the resolver. The latter is a stable slice of Hackage, the package base for Haskell which guarantees that all packages are built and get on with each other (NB: such slices are very much needed in other languages): ).

stack.yaml:

resolver: lts-17.11

packages:
- hello-world

You can find the recipe for building the specific package in hello-world.cabal:

cabal-version:      2.4
name:               hello-world
version:            1.0
synopsis:           Hello World
license:            MIT
license-file:       LICENSE
author:             Nick

executable hello-world
    main-is:          Main.hs
    build-depends:    base >= 4 && < 5
    hs-source-dirs:   src
    default-language: Haskell2010
    ghc-options:      -Wall -O2

Our Nix code will use these two files to find the information on what exactly and how it should build. By way of an experiment, we can check if our code really runs and works:

$ stack run hello-world
Hello, World!

Come to the dar^Wnix side, we have cookies!

Though Stack by itself is a great tool used to build projects in Haskell, it lacks many capabilities. There is the library haskell.nix developed by IOHK to build Haskell programs for Nix. This is what we’re going to use here. To begin with, let's make sure that our project is built using Nix.

Haskell.nix allows using several lines to convert all data on our project build from .cabal-files and stack.yaml into derivationfor Nix.

nix/stackyaml.nix:

{
  # We import the latest version of haskell.nix from GitHub and use it to initialize Nixpkgs.
  haskellNix ? import (builtins.fetchTarball "https://github.com/input-output-hk/haskell.nix/archive/b0d03596f974131ab64d718b98e16f0be052d852.tar.gz") {}
  # Here we use the latest stable version of Nixpkgs. Version 21.05 is coming soon :)
, nixpkgsSrc ? haskellNix.sources.nixpkgs-2009
, nixpkgsArgs ? haskellNix.nixpkgsArgs
, pkgs ? import nixpkgsSrc nixpkgsArgs
}:

let
  # We create a project based on stack. Function cabalProject is available for Cabal projects.
  project = pkgs.haskell-nix.stackProject {
    name = "hello-world";

    # Derivation with the project source code.
    # To build the project, function cleanGit copies only the files available in our git-repository.
    src = pkgs.haskell-nix.haskellLib.cleanGit {
      name = "hello-world";

      # src parameter must refer to the root directory containing stack.yaml.
      src = ../.;

      # keepGitDir keeps .git directory when creating the build.
      # This can be of use, for example, to insert the commit hash into the code.
      keepGitDir = true;
    };

    # You can indicate the build parameters in the modules parameter either for all modules at once or for each module individually.
    modules = [{
      # doCheck is responsible for running unit tests when building the project, including the ones contained in all dependencies.
      # Here we want to avoid this, which is why the best thing is to define this parameter as false and enable it only for the required
      # packages.
      doCheck = false;

      # Let’s add -Werror flag for our Hello World.
      packages.hello-world.components.exes.hello-world.ghcOptions = [ "-Werror" ];
    }];
  };

# To the outside of this file we set out the project -- our project, and pkgs -- nixpkgs slice we’ll use further.
in { inherit project; inherit pkgs; }

Let’s check whether it’s now possible to build our project using Nix. To do this, it’s enough to use nix build command. As usual, the symbolic link result containing the resulting build will be created in the current directory.

$ nix build project.hello-world.components.exes
$ ./result/bin/hello-world
Hello, World!

Great! The little magic shown above has provided a completely reproducible build of our project, including all system dependencies. Let’s keep going!

Dockerfile? What Dockerfile?

It’s 2021 and many companies use Docker to deploy and launch services. Typeable is not an exception here. nixpkgs include very convenient tools named dockerTools designed to build containers. For more details on their capabilities open the link. I’m just going to show how we pack our code in containers using these tools. You can see the complete code in the file nix/docker.nix.

For a start, we’ll need the source container where we’ll place everything we need. Nix allows building a container completely from scratch without any unneeded components, but this approach is not always convenient. Sometimes, especially in abnormal situations, you have to get into the container manually using the command line. That’s why we use CentOS here.

sourceImage = dockerTools.pullImage {
  imageName = "centos";
  imageDigest = "sha256:e4ca2ed0202e76be184e75fb26d14bf974193579039d5573fb2348664deef76e";
  sha256 = "1j6nplfs6999qmbhjkaxwjgdij7yf31y991sna7x4cxzf77k74v3";
  finalImageTag = "7";
  finalImageName = "centos";
};

For those who’ve ever worked with Docker everything is obvious here. We tell Nix what image from the public Docker Registry we’d like to use and specify that from now on we’ll refer to it as sourceImage.

The image itself is built using the function buildImage available in the dockerTools. This function has quite a few parameters and it’s often easier to write a wrapper for it, which is exactly what we’re going to do:

makeDockerImage = name: revision: packages: entryPoint:
  dockerTools.buildImage {
    name = name;
    tag = revision;
    fromImage = sourceImage;
    contents = (with pkgs; [ bashInteractive coreutils htop strace vim ]) ++ packages;
    config.Cmd = entryPoint;
  };

Our function makeDockerImage accepts four parameters: the container name, its version (at Typeable, we usually use a commit hash from git as the tag), the packages we’d like to enable and the entry point used at the container startup. Inside it we refer to the CentOS image as the basis (fromImage), plus add some utilities which can be extremely useful in emergencies.

Finally, let's create the image with our marvellous application.

hello-world = project.hello-world.components.exes.hello-world;
helloImage = makeDockerImage "hello"
  (if imageTag == null then "undefined" else imageTag)
  [ hello-world ]
  [ "${hello-world}/bin/hello-world"
  ];

As the first step, we create an alias for the package we need so that we don’t need to write project.hello-world... everywhere. After that, we create the image of the container with hello-world package by calling the function makeDockerImage created previously. Parameter imageTag passed from the outside will be indicated as the tag; if nothing was passed, “undefined” will be indicated.

Let's check the build:

$ nix build --argstr imageTag 1.0 helloImage
[4 built, 0.0 MiB DL]

 $ ls -l result
lrwxrwxrwx 1 user users 69 May 11 13:12 result -> /nix/store/56qqhiwahyi46g6mf355fjr1g6mcab0b-docker-image-hello.tar.gz

In a couple of minutes or even sooner we’ll get the symbolic link result leading to our finished image. Check again whether everything works as intended.

$ docker load < result
76241b8b0c76: Loading layer [==================================================>]  285.9MB/285.9MB
Loaded image: hello:1.0
$ docker run hello:1.0
Hello, World!

Conclusion

The result is that we’ve managed to create a reproducible build of our Haskell project using a comparatively small amount of code. The same can be done for projects in other languages after replacing haskell.nix with something else: nixpkgs has built-in tools for C/C++, Python, Node and other popular languages.

In the next post, I’ll tell you about the frequent issues occurring when you work with Nix. Stay tuned!