TL;DR: Containers enable DevOps from prototype to production at extreme scale; here experiences from DoE’s Sandia National Laboratories are shared.

SUG Series Introduction

The inaugural meeting of the Singularity User Group (SUG) was held March 12-13, 2019, at the San Diego Supercomputer Center (SDSC). The event attracted diverse representation from the international advanced computing community as conveyed through the post-event press release issued jointly by SDSC and Sylabs.

Over the course of the two-day event, over 20 talks were presented by members of the Singularity user, developer, and provider community. Because SUG generated a significant amount of interest, even from those who were unable to attend, we are sharing online each of the talks presented.

SUG Talk Introduction

Andrew Younge, a Computer Scientist at Sandia National Laboratories, makes the case here for enabling DevOps across heterogeneous HPC platforms via containers – enablement that “… could improve overall build and testing efficiency for many teams.” He then applies enabling containers at increasingly larger scales. The abstract for Andrew’s contributed talk entitled From Containerizing Testbeds for HPC Applications to Exascale Supercontainers is as follows:

As the code complexity of HPC applications expand, development teams continually rely upon detailed software operation workflows to enable automation of building and testing their application. These development workflows can become increasingly complex and, as a result, difficult to maintain when the target platforms’ environments are increasing in architectural diversity and continually changing. Recently the advent of containers in industry have demonstrated the feasibility of such workflows, and the latest support for containers in HPC environments makes them now attainable for application teams. Fundamentally, containers have the potential to provide a mechanism for simplifying workflows for development and deployment, which could improve overall build and testing efficiency for many teams.

This talk will provide an overview of the current strategy for utilizing containers in HPC, starting at Sandia and expanding to the broader DOE Exascale ecosystem. This first includes the initial efforts for running Singularity on testbed clusters, expanding to Singularity deployments on DOE/NNSA production supercomputing resources. From here, the ECP Supercomputing Containers Project (aka ECP Supercontainers) will be introduced, which represents a consolidated effort across the DOE and NNSA to use a multi-level approach to accelerate adoption of container technologies. This project will investigate container scalability, interoperability, and ensure a high level of integration future HPCsystems and application teams. Effectively, we hope to ensure container runtimes like Singularity will be well poised to take advantage of the first Exascale supercomputers across the DOE.

Andrew’s talk from SUG can be found below and here. Enjoy!