Research Software - Singularity

Singularity

Sep 28, 2020. | By: @vsoch

Do you remember the reproducibility crisis? It’s a term so well-coined that it’s on Wikipedia, and although some of us possibly didn’t have awareness for it until 2014 or 2015, the term has been thrown around already for almost a decade. For the software showcase, we’ve featured workflow managers quite a bit, and it naturally must come to follow that we also feature container technologies, which often are the unit of operation passed around by the manager. In celebration of Google HPC Cloud Days featuring this container technology this week, the RSEPedia software showcase is proud to present Singularity for the showcase this week. Singularity, since 2015, has really taken the high performance computing, especially academic, world by storm by providing a means to run trusted containers on large research clusters with many users. It’s not the only, or a sure fire way to have reproducible science, but it sure is a good contender solution! Keep reading the following sections to learn more!

/rseng/assets/img/posts/showcase/singularity.png

If you are already a container nerd, we encourage you to contribute to the research software encyclopedia and annotate the respository:

otherwise, keep reading!

What is Singularity
How do I cite it?
How do I contribute to the software survey
Where can I learn more?

What is Singularity?

Singularity is a container technology that allows you to package up your operating system, libraries, and special sauce software into a read only binary that can be shared with your colleagues to reproduce your work. If you’ve never used a container before, it sort of feels like shelling into another computer (akin to using ssh) but it’s in fact a container instance running on your machine. Singularity offers:

instead of using layers like Docker, a single, single-file based container format that can be signed
an ability to work seamlessly with traditional HPC technologies like MPI and filesystems
the same entrypoints you would expect from Docker like exec and run, but also shell.
an ability to convert from a Docker image right on the fly with the docker:// unique resource identifier
seamless environment with the host as the usual default, unless you specify otherise
a large, knowledgeable community.

Did you know that…

While some of this is old news, here are some fun factoids that you might not have known about Singularity.

Founding

Singularity was created by Gregory Kurzer at Berkeley Lab - although unfortunately the punk rock looking site design is not preserved, you can see the original content on the wayback machine (take a look at the favicon to see a glimpse of the branding). It started on his personal GitHub, and was eventually encouraged to be moved to a more community driven organization, Singularityware, which is no longer used. The code base has since moved twice, from Sylabs and now to hpcng. Where will it fly to next?

Languages

Did you know that the original Singularity implementation was done in bash, C, and eventually Python? C handled low level routines, Python handled higher level API communication and parsing functions, and bash tied it all together. It wasn’t until Singularity went under the umbrella of Sylabs that it was fully ported to Go, which of course adds a lot of benefits over it’s previously very-academically appropriate hodge-podge of languages. Take a look at the 2.x branch for a walk down history lane!

Docker

Singularity didn’t always play nicely with Docker, and in fact it was a contentious feature when it first came under discussion! An early developer (this writer, @vsoch) advocated for a designed the feature, and added the feature now over four years ago. Many features that you have come to appreciate and enjoy that make it feel like Docker came directly from this same route.

Forgotten Features

Did you know that an early version of Singularity had a feature called checks? Added in summer of 2017, the idea was that the user could run singularity check <container.sif> and a series of automated security or other linting checks would be run, of course under control of the sysadmin managing the install. Although this feature seemed useful to the developer that implemented it, it was not ported to the Go version of Singularity and is since lost. However, on a positive note, the scientific filesystem, called Singularity apps, a way to build a container with multiple entrypoints, was carried forward.

Do you have any cool, historical facts to share about Singularity? Let us know!. Otherwise, if reproducibility is the name of your game, or if you’ve never tried it before, you should check out Singularity!

How do I cite it?

You can cite the paper from PLOS One.

@article{10.1371/journal.pone.0177459,
    author = {Kurtzer, Gregory M. AND Sochat, Vanessa AND Bauer, Michael W.},
    journal = {PLOS ONE},
    publisher = {Public Library of Science},
    title = {Singularity: Scientific containers for mobility of compute},
    year = {2017},
    month = {05},
    volume = {12},
    url = {https://doi.org/10.1371/journal.pone.0177459},
    pages = {1-20},
    abstract = {Here we present Singularity, software developed to bring containers and reproducibility to scientific computing. Using Singularity containers, developers can work in reproducible environments of their choosing and design, and these complete environments can easily be copied and executed on other platforms. Singularity is an open source initiative that harnesses the expertise of system and software engineers and researchers alike, and integrates seamlessly into common workflows for both of these groups. As its primary use case, Singularity brings mobility of computing to both users and HPC centers, providing a secure means to capture and distribute software and compute environments. This ability to create and deploy reproducible environments across these centers, a previously unmet need, makes Singularity a game changing development for computational science.},
    number = {5},
    doi = {10.1371/journal.pone.0177459}
}

There is an earlier paper from 2012 as well.

How do I get started?

The Sylabs Documentation is a good place to start
Google HPC Days is featuring a tutorial later this week!
The Singularity Slack has lots of folks to talk to
The Singularity Google Group is a good place to ask for help.

Generally, if you do a GitHub search for Singularity tutorial or getting started, you’ll get a robust set of resources, as many centers have rolled their own getting started guides. You can also post questions on the GitHub issues board.

How do I contribute to the software survey?

or read more about annotation here. You can clone the software repository to do bulk annotation, or annotation any repository in the software database, We want annotation to be fun, straight-forward, and easy, so we will be showcasing one repository to annotate per week. If you’d like to request annotation of a particular repository (or addition to the software database) please don’t hesitate to open an issue or even a pull request.

Where can I learn more?

You might find these other resources useful:

The Research Software Database on GitHub
RSEpedia Documentation
Google Docs Manuscript you are invited to contribute to.
Annotation Documentation for RSEpedia
Annotation Tutorial in RSEng docs

For any resource, you are encouraged to give feedback and contribute!