Maintaining Open Source Singularity Since 2017

SCS Login

scs-build: Singularity build automation tool

By Staff

Mar 11, 2022 | Blog, SingularityCE Updates

scs-build is a new command-line tool created by Sylabs that can be used to invoke remote builds of SIF images through Sylabs Cloud or an on-premises Singularity Enterprise installation.

Using only your access token obtained through the Sylabs Cloud/Singularity Enterprise frontend, you can now remotely build a SIF file that is (optionally) automatically pushed to the Cloud Library or can be downloaded locally.

Also distributed as an OCI compatible container image, scs-build is ready for integration into any automated CI/CD workflow supporting OCI container images. Even without an automated CI/CD DevOps platform, scs-build can be invoked through a cronjob to automate builds of SIF images.

Example Workflows

Create builds from your workstation

Use an OCI compatible runtime (Docker, podman, etc.) to invoke scs-build. This is especially useful when running on a workstation (ie. Mac) that does not have the ability to run singularity.

export SYLABS_AUTH_TOKEN=xxx
docker run --rm \
    -e SYLABS_AUTH_TOKEN=${SYLABS_AUTH_TOKEN} \
    sylabsio/scs-build \
    build docker://alpine:3 library:<myuser>/default/alpine:3

or download SIF artifact locally:

The following example will download the SIF artifact in the current working directory.

docker run -u $(id -u) -v `pwd`:/work --rm \
    -e SYLABS_AUTH_TOKEN=${SYLABS_AUTH_TOKEN} \
    sylabsio/scs-build \
    build docker://alpine:3 /work/alpine_3.sif

Create Base images for your application SIFs

Use scs-build in a CI/CD workflow to build “pristine” base images containing operating system of choice and required base packages and/or configuration.

For example, the following DEF file can be used to create a Ubuntu “focal” base SIF image:

# ubuntu_base.def

Bootstrap: docker
From: registry.mydomain.com/ubuntu:focal

%post
  export DEBIAN_FRONTEND=noninteractive

  # update package index
  apt-get upgrade

  # install latest packages
  apt-get --quiet --assume-yes update

  # add vim
  apt-get install --quiet --assume-yes vim

  # clean up a bit
  apt-get clean

Use Sylabs Cloud Remote Build

export SYLABS_AUTH_TOKEN=xxx    # obtained through Sylabs Cloud web UI
scs-build build ubuntu_base.def library:<myuser>/default/ubuntu:focal

Build using your local Singularity Enterprise install

export SYLABS_URL=https://cloud.mydomain.com
export SYLABS_AUTH_TOKEN=xxx
scs-build build ubuntu_base.def library:library/default/ubuntu:focal

When using a “strategic” library reference library:library/default/ubuntu:focal, the base image can then be referenced in derivative images as follows:

singularity shell library://ubuntu:focal

Note: this only works with an “admin” token on Singularity Enterprise. Regular Sylabs Cloud users do not have write access to the project named “library”.

Create a build that uses the “base” image

Create a DEF file for the “derivative” image:

# derivative.def

Bootstrap: library
Library: https://library.mydomain.com    # omit this line if using Sylabs Cloud
From: ubuntu:focal    # change this to "<myuser>/default/ubuntu:focal" if using Sylabs Cloud

%post
  export DEBIAN_FRONTEND=noninteractive

  apt-get update && apt-get install --quiet --assume-yes vim-pathogen

  apt-get --quiet --assume-yes clean

Build derivative SIF using scs-build:

export SYLABS_URL=https://cloud.mydomain.com
export SYLABS_AUTH_TOKEN=xxx
scs-build build derivative.def library:myuser/custom/image

The build artifact can then be pulled/executed as follows:

singularity pull custom.sif library://myuser/custom/image:latest

or…

singularity shell library://myuser/custom/image:latest

CI/CD Workflows

We have also provided example configuration files for both GitHub Actions and GitLab that can be used as a starting point for integrating this into your existing CI/CD workflows.

Join Our Mailing List

Categories

Related Posts

An Introduction to Singularity Containers

Mar 30, 2023 |

Enabling Portable and Secure Computing Environments for High-Performance Workloads.As part of their ongoing efforts to streamline workflows, enhance productivity, and save time, engineers, and developers in enterprises and high performance computing (HPC) focused...

read more