The hybrid workflow of Singularity with Docker containers combines the strengths of both technologies to provide a flexible and efficient containerization solution. With Singularity’s ability to natively run Docker containers, users can leverage the vast Docker ecosystem and easily incorporate Docker images into their Singularity workflows. This hybrid approach allows users to benefit from Singularity’s security, reproducibility, and compatibility with HPC systems while taking advantage of Docker’s extensive collection of pre-existing containers and tools. It provides a seamless bridge between the two container technologies, enabling users to run Docker containers within Singularity with minimal effort and enabling efficient collaboration across diverse computing environments. As work progresses towards SingularityCE 4.0 and beyond, let’s take a look at what is possible today with Singularity Enterprise and SingularityCE/PRO.
Docker and a brief OCI history: While Docker initially had its own container format and runtime, it actively collaborated with the Open Container Initiative (OCI)
to align its technology with the industry standards. As a result, Docker adopted the OCI specifications as the foundation for its container format and runtime. Docker now utilizes the OCI image and runtime specifications, ensuring compatibility and interoperability with other OCI-compliant container runtimes and tools.
Singularity and the SIF format: With Singularity, the container image is a single file with a .sif extension, which contains a root filesystem in SquashFS format. This file can be easily distributed and executed on different systems running Singularity, making it portable across various environments. One advantage of using the SquashFS format is that Singularity containers can be mounted as read-only, providing enhanced security and reproducibility. Additionally, Singularity containers can directly access the host system’s file system, allowing users to work with data and files seamlessly without the need for complex data volume setups.
Hybrid Container Workflows: OCI powers cloud native technologies such as Docker and Kubernetes, which have changed the way the world develops and deploys software. Containers, such as those provided by Singularity, allow users to package their software into a single, reproducible unit, making it easy to deploy and run the application on HPC environments. Singularity is capable of using OCI containers as a source to either run the container or convert it to a SIF image. And while there has always been a high level of compatibility, it has never been a story of perfect compatibility. We’ve been talking to users in the open source community, as well as our customers, and partners to understand current and future needs. These conversations spark our imaginations to think a lot about what OCI workflows look like in HPC environments today, how they will evolve tomorrow. More importantly, we think about security (we’ll get to more of that in other blog posts) and what it will take to simplify interactions with the merging orchestrated and batch computing workflows.
In this article, we’ll look at a sample “hybrid” OCI/Singularity workflow that leverages Docker to build, test and push an image to Singularity Enterprise using the OCI standards, and then pull and run this image with both Docker and SingularityCE/PRO.
Closing the gap
Singularity Enterprise and Singularity Container Services now include functionality to store OCI images and artifacts, in addition to SIF images. This means that we can build an image using a Dockerfile, upload it to the Library, and run it using SingularityCE/PRO.
Let’s take the following Dockerfile as an example:
When running this image, the output is:
SingularityCE and PRO normally push images to the Library at either your private Singularity Enterprise endpoint or the SCS endpoint at cloud.sylabs.io. OCI native tools like Docker cannot communicate using these endpoints so we have created a new OCI compatible Library at registry.sylabs.io.
Before pushing this image to the Library, we must configure Docker to push the image with tags. If your Singularity installation (SingularityCE or SingularityPRO) doesn’t have the OCI registry remote environment configuration, use the following example to add the capability.
Then, obtain a password to authenticate to the built-in registry and pipe the output to docker login command:
Once you are logged in, we must re-tag the image so Docker knows where to store the image, then, push it.
This image will now be visible and classified as an OCI / Docker image, this new asset type is assigned an icon and a type field specified as DOCKER.
Continuing with the workflow, let’s run this image from Singularity as an OCI image, we will follow the same steps, obtain a password to authenticate to the registry and then run the image with the docker:// schema.
Reuse of OCI image as a base for subsequent use is also possible, in your SingularityCE definition files, let’s add the following code to a file named example.def:
And finally issue the SingularityCE build command:
Singularity Enterprise 2.3+ and Singularity Container Service allows more flexibility than ever to work with the increasing number of tools in the OCI ecosystem. Whether it’s building and testing container images with Docker, Kubernetes, SingularityPRO, or any other tools leveraging the OCI standards, Singularity Enterprise and Singularity Container Service provides a single place to store and manage your container images and OCI artifacts.
In this blog post, we’ve demonstrated a workflow that starts off by building an image in Docker, and can be run by SingularityCE/PRO in an efficient and seamless way in an HPC environment in its original OCI format.
Stay tuned for more blog posts to come that look at other ways to use the new OCI compatibility built into Singularity Enterprise 2.3 and Singularity Container Service. If you have any questions or comments about hybrid OCI/Singularity workflows, feel free to contact us. We’d love to hear from you!