scs-build: Singularity build automation tool

By Staff

Mar 11, 2022 | Blog, SingularityCE Updates

scs-build is a new command-line tool created by Sylabs that can be used to invoke remote builds of SIF images through Sylabs Cloud or an on-premises Singularity Enterprise installation.

Using only your access token obtained through the Sylabs Cloud/Singularity Enterprise frontend, you can now remotely build a SIF file that is (optionally) automatically pushed to the Cloud Library or can be downloaded locally.

Also distributed as an OCI compatible container image, scs-build is ready for integration into any automated CI/CD workflow supporting OCI container images. Even without an automated CI/CD DevOps platform, scs-build can be invoked through a cronjob to automate builds of SIF images.

Example Workflows

Create builds from your workstation

Use an OCI compatible runtime (Docker, podman, etc.) to invoke scs-build. This is especially useful when running on a workstation (ie. Mac) that does not have the ability to run singularity.

export SYLABS_AUTH_TOKEN=xxx
docker run --rm \
    -e SYLABS_AUTH_TOKEN=${SYLABS_AUTH_TOKEN} \
    sylabsio/scs-build \
    build docker://alpine:3 library:<myuser>/default/alpine:3

or download SIF artifact locally:

The following example will download the SIF artifact in the current working directory.

docker run -u $(id -u) -v `pwd`:/work --rm \
    -e SYLABS_AUTH_TOKEN=${SYLABS_AUTH_TOKEN} \
    sylabsio/scs-build \
    build docker://alpine:3 /work/alpine_3.sif

Create Base images for your application SIFs

Use scs-build in a CI/CD workflow to build “pristine” base images containing operating system of choice and required base packages and/or configuration.

For example, the following DEF file can be used to create a Ubuntu “focal” base SIF image:

# ubuntu_base.def

Bootstrap: docker
From: registry.mydomain.com/ubuntu:focal

%post
  export DEBIAN_FRONTEND=noninteractive

  # update package index
  apt-get upgrade

  # install latest packages
  apt-get --quiet --assume-yes update

  # add vim
  apt-get install --quiet --assume-yes vim

  # clean up a bit
  apt-get clean

Use Sylabs Cloud Remote Build

export SYLABS_AUTH_TOKEN=xxx    # obtained through Sylabs Cloud web UI
scs-build build ubuntu_base.def library:<myuser>/default/ubuntu:focal

Build using your local Singularity Enterprise install

export SYLABS_URL=https://cloud.mydomain.com
export SYLABS_AUTH_TOKEN=xxx
scs-build build ubuntu_base.def library:library/default/ubuntu:focal

When using a “strategic” library reference library:library/default/ubuntu:focal, the base image can then be referenced in derivative images as follows:

singularity shell library://ubuntu:focal

Note: this only works with an “admin” token on Singularity Enterprise. Regular Sylabs Cloud users do not have write access to the project named “library”.

Create a build that uses the “base” image

Create a DEF file for the “derivative” image:

# derivative.def

Bootstrap: library
Library: https://library.mydomain.com    # omit this line if using Sylabs Cloud
From: ubuntu:focal    # change this to "<myuser>/default/ubuntu:focal" if using Sylabs Cloud

%post
  export DEBIAN_FRONTEND=noninteractive

  apt-get update && apt-get install --quiet --assume-yes vim-pathogen

  apt-get --quiet --assume-yes clean

Build derivative SIF using scs-build:

export SYLABS_URL=https://cloud.mydomain.com
export SYLABS_AUTH_TOKEN=xxx
scs-build build derivative.def library:myuser/custom/image

The build artifact can then be pulled/executed as follows:

singularity pull custom.sif library://myuser/custom/image:latest

or…

singularity shell library://myuser/custom/image:latest

CI/CD Workflows

We have also provided example configuration files for both GitHub Actions and GitLab that can be used as a starting point for integrating this into your existing CI/CD workflows.

Join Our Mailing List

Related Posts

QA and Stability in Singularity

There are many different approaches that can be taken when building software. At one end of the spectrum is the extreme caution and conservatism that’s appropriate, for example, of safety critical code used in vehicles or in real-time operating systems. At the other...

read more

Improve Security in your CI/CD Workflows

In the development world, continuous integration is where members of a team integrate all their work frequently, for example, think of a team all working on the same code base, they are fixing bugs, implementing new features, so to prevent conflicts, all the code is...

read more