Singularity Container Workflow: Part 4 – Signing the Container

By Staff

Aug 25, 2022 | Blog, How To Guides

Signing the Container

The Singularity 3.0 family introduced the ability to create (and manage) PGP keys to sign and verify containers. This provides a trusted method for Singularity users to share containers and ensures a bit-for-bit reproduction of the original container as the author intended.

For security purposes, it is important that private keys remain private. Singularity Container Services only stores your public key information. In this example, pull the TensorFlow container previously created, sign it, then store the signed container in the SCS Library.

Generating and managing PGP keys

To sign your containers you first need to generate one or more keys with Singularity CLI, which can then be pushed to the SCS Keystore.

Create a new keypair by entering the email address associated with our Singularity Container Services account.

We’ll then enter our account details, add a comment and passphrase (which will not be displayed) and have the Singularity client push it to the keystore.

$ singularity key newpair
Enter your name (e.g., John Doe) : John Doe
Enter your email address (e.g., :
Enter optional comment (e.g., development keys) : scs demo keys
Enter a passphrase :
Would you like to push it to the keystore? [Y,n] y
Generating Entity and OpenPGP Key Pair... done
Key successfully pushed to keystore

With that key created, let’s use the command line and web interface to display the keys.

Listing keys

$ singularity key list
Public key listing (/home/demo/.singularity/sypgp/pgp-public):

0) U: John Doe (scs demo keys) <>
   C: 2022-07-27 13:23:15 -0700 PDT
   F: ED0538B8C7B503018BEA03D8B35109D3EE6937F4
   L: 4096

In the output above the index of my key is 0 and the letters stand for the following:

U: User

C: Creation date and time

F: Fingerprint

L: Key length

*for more information on signing, please see the Singularity Signing and Verify documentation

From the SCS Dashboard, we’ll select “My Keys”. Listed below is the key we had created via the SIngularity client and pushed to the SCS Keystore. We can see the Key name and description, creation date, algorithm and bits used in the creation, followed by the fingerprint. From here we also have examples of a container signing and image verification commands.

With verification of the access token, the next step is to check the status of the services.

Pull then Sign the container

Because SCS does not store your private key information, containers must be signed by the local Singularity client where your private key information resides. There are two methods to retrieve a container from the SCS Library Pull a container from the Library using the Singularity client “pull” command Download the container from the SCS web interface Let’s pull the unsigned Tensorflow container we previously created. In this example, we will work with the command line.
$ singularity pull library://sylabsdemo/containers/tensorflow:latest-gpu

INFO:    Downloading library image
2.6GiB / 2.6GiB [==================================================================================] 100 % 10.9 MiB/s 0s
WARNING: integrity: signature not found for object group 1
WARNING: Skipping container verification

*When pulling the container with a tag, you may notice the semicolon is converted to underscore in the resulting file name.

$ singularity sign tensorflow_latest-gpu-signed.sif
Signing image: tensorflow_latest-gpu.sif
Enter key passphrase :
Signature created and applied to tensorflow_latest-gpu.sif

The first time you sign a container, Singularity will generate a PGP signing keypair and optionally upload the public key to Singularity Container Services Keystore, where it will be available for container verification requests. 

Enter the passphrase key you created during the “singularity key newpair” command and you will see the container has been signed. For advanced signing capabilities, such as signing objects withing the SIF, please see Singularity SIF ID’s and Groups documentation.

You can see the details of the signed container by using the singularity sif list command:

$ singularity sif list tensorflow_latest-gpu-signed.sif
ID   |GROUP   |LINK    |SIF POSITION (start-end)  |TYPE
1    |1       |NONE    |32768-32826               |Def.FILE
2    |1       |NONE    |36864-39757               |JSON.Generic
3    |1       |NONE    |40960-41660               |JSON.Generic
4    |1       |NONE    |45056-2749640704          |FS (Squashfs/*System/amd64)
5    |NONE    |1   (G) |2749640704-2749642501     |Signature (SHA-256)

Singularity SIF Verification

Use the SIF verification functionality to ensure the local container has not been modified:

$ singularity verify tensorflow_latest-gpu-signed.sif
Verifying image: tensorflow_latest-gpu.sif
[LOCAL]   Signing entity: John Doe (scs demo keys) <>
[LOCAL]   Fingerprint: ED0538B8C7B503018BEA03D8B35109D3EE6937F4
Objects verified:
1   |1       |NONE    |Def.FILE
2   |1       |NONE    |JSON.Generic
3   |1       |NONE    |JSON.Generic
4   |1       |NONE    |FS
Container verified: tensorflow_latest-gpu.sif

If another user ran this verification against an image in SCS Library, Singularity would retrieve the public key from the Keystore automatically, rather than using [LOCAL]

Push to the Library

Okay – let’s go ahead and push our newly signed Tensorflow container up to the Library, with the Singularity push command, adding a new tag name:

$ singularity push tensorflow_latest-gpu-signed.sif library://sylabsdemo/containers/tensorflow:latest-gpu-signed

2.6GiB / 2.6GiB [=====================================================================================================] 100 % 2.6 MiB/s 0s. 

Now that the container is pushed up to the SCS Library, we’ll take a quick look at the repository. We can see both signed and unsigned versions are available. And it is through the web interface that we can add tags or delete tags. If you want to change a tag, add a new one and remove the old. 

*See the Singularity Documentation: Pushing a Container for more information on tags

By default, Singularity Container Services repositories are private, unless made public. In our case, we can make this project public. To show this, we will open another WSL2 (Debian) with Singularity installed using the default settings. By issuing a “pull” command we can see the container is not accessible. If we go back into the web interface and make this a public project, we can then see the container is now available for public “pulls”.

*If you tried pulling a container from the Library that is not public, you’d receive the following error message

$ singularity pull library://sylabsdemo/containers/tensorflow:latest-gpu-signed

FATAL:   While pulling library image: error fetching image: request did not succeed: UNAUTHORIZED: unauthorized to access repository: sylabsdemo/containers/tensorflow, action: pull: unauthorized to access repository: sylabsdemo/containers/tensorflow, action: pull (401 Unauthorized)


We have taken what we’ve done in the past and continued on by signing our TensorFlow container, verifying that container was unchanged, and pushing that container to the Singularity Container Services Library.

In the next installment, we will view and search for containers through the Singularity Container Services interface as well as perform some advanced Singularity functions such as encryption and inspection of SIF content.

Thank you for joining us!

Join Our Mailing List

Related Posts