Docker and Singularity are container systems that allow users full control over their environment. You can create your own container image which your job will execute within, or choose from a set of pre-defined images. For more information about Docker, please see:
and Singularity, please see:
For jobs on OSG, it does not matter whether you provide a Docker or Singularity image. Either is compatible with our system and can be used with little to no modification. This is of course highly dependent on your workload. Please feel free to contact us if you have any questions.
Singularity has become the preferred containerization method in scientific computing. The following talk describes Singularity for scientific computing:
Derek Weitzel wrote a blog post about Singularity on OSG, which provides a good introduction on how to create images and run them, but does not cover all the functionality described further down:
The default setup is to auto load an image on sites which support Singularity. Every
job which lands on such a site, will have a container started just for that job, and
then run within that container. Most users will not even know that their jobs are run
within a container, but it will provide them with a consistent environment across
OSG sites. The current default container is based on EL6 and contains a basic
set of tools expected from OSG compute nodes. The image is loaded from
/cvmfs/singularity.opensciencegrid.org/opensciencegrid/osgvo-el6 and the definition file
is available in GitHub
If you want to steer a job to run on a default Singularity instance,
HAS_SINGULARITY == True in the job requirements. For example:
universe = vanilla executable = job.sh Requirements = HAS_SINGULARITY == TRUE should_transfer_files = IF_NEEDED when_to_transfer_output = ON_EXIT output = out error = err log = log queue
To instruct the system to load a different image, use the
+SingularityImage attribute in
your job submit file. For example, to run your job under EL7:
universe = vanilla executable = job.sh Requirements = HAS_SINGULARITY == TRUE +SingularityImage = "/cvmfs/singularity.opensciencegrid.org/opensciencegrid/osgvo-el7:latest" +SingularityBindCVMFS = True should_transfer_files = IF_NEEDED when_to_transfer_output = ON_EXIT output = out error = err log = log queue
The user support team maintains a set of images. These contain a basic set of tools and libraries. The images are are:
|EL 6||/cvmfs/singularity.opensciencegrid.org/opensciencegrid/osgvo-el6:latest||GitHub||A basic Enterprise Linux (CentOS) 6 based image. This is currently our default image|
|EL 7||/cvmfs/singularity.opensciencegrid.org/opensciencegrid/osgvo-el7:latest||GitHub||A basic Enterprise Linux (CentOS) 7 based image.|
|Ubuntu Xenial||/cvmfs/singularity.opensciencegrid.org/opensciencegrid/osgvo-ubuntu-xenial:latest||GitHub||A good image if you prefer Ubuntu over EL flavors|
|Ubuntu 18.04 (Bionic)||/cvmfs/singularity.opensciencegrid.org/opensciencegrid/osgvo-ubuntu-18.04:latest||GitHub||A good image if you prefer Ubuntu over EL flavors|
|TensorFlow||/cvmfs/singularity.opensciencegrid.org/opensciencegrid/tensorflow:latest||GitHub||Base on the TensorFlow base image, with a few OSG package added|
|TensorFlow GPU||/cvmfs/singularity.opensciencegrid.org/opensciencegrid/tensorflow-gpu:latest||GitHub||Used for running TensorFlow jobs on OSG GPU resources|
Exploring Images on the Submit Host
Images can be explored interactively on the submit hosts by starting it in "shell" mode. The recommended command line, similar to how containers are started for jobs, is:
singularity shell \ --home $PWD:/srv \ --pwd /srv \ --bind /cvmfs \ --scratch /var/tmp \ --scratch /tmp \ --contain --ipc --pid \ /cvmfs/singularity.opensciencegrid.org/opensciencegrid/osgvo-ubuntu-xenial:latest
Frequently Asked Questions / Common Issues
FATAL: kernel too old
If you get a FATAL: kernel too old error, it means that the glibc version in the image is too new for the kernel on the host. You can work around this problem by specifying the minimum host kernel. For example, if you want to run the Ubuntu 18.04 image, specfy a minimum host kernel of 3.10.0, formatted as 31000 (major * 10000 + minor * 100 + patch):
Requirements = HAS_SINGULARITY == True && OSG_HOST_KERNEL_VERSION >= 31000
OSG Connect provides tooling for users to create, publish and load custom images. This is useful if your job requires some very specific software setup.
Creating a Custom Image
If you want to use an image you have created yourself, the image
should be defined as a Docker image and published in the Docker
Hub. The reason we use Docker as a source
image repository is that it allows us to easily import the images into
our own distribution system (see below). To get started, create a Docker
user, sign in to the hub, and create a new repository. You will end up
with an identifier of the
There are two main methods, we recommend to generating your own custom image:
- Editing the
- Editing the default image using local Docker
which are described below. Once you have created your custom image,
you will need to register the image as described in the next section.
If you prefer, you can base you image on images not already published
by OSG, but if you do this, we recommend that you as one of the steps
/cvmfs directory. This will enable the container to access
tools and data published on
/cvmfs. If you do not want
in the container, please add
+SingularityBindCVMFS = False to your job submit file.
To create an image locally using a
first download the
Dockerfile for your desired flavor from
OSG GitHub repository,
see table above. For example, if you want to base the image on our Ubuntu
$ git clone https://github.com/opensciencegrid/osgvo-ubuntu-xenial.git
and change into the directory
$ cd osgvo-ubuntu-xenial
Dockerfile in the directory to fit your requirements. If
/cvmfs mounted in your image please add
# required directories RUN mkdir -p /cvmfs
Then build the image with tag matching your Docker Hub repository:
$ docker build -t namespace/repository_name .
Once you have a successful build, push it to the hub:
$ docker push namespace/repository_name
Editing the default image using local Docker
To create an image locally using Docker, you will need to have Docker installed on your machine. For install instructions, see the Docker documentation.
With docker installed you can now get the desired image from Docker Hub. Again, we will look at OSG Ubuntu Xenial image.
$ docker pull opensciencegrid/osgvo-ubuntu-xenial
We will run the image in a docker interactive session
$ docker run -it --name <docker_session_name_here> opensciencegrid/osgvo-ubuntu-xenial /bin/bash
Giving the session a name is important because it will make it easier to reattach the session later and commit the changes later on. Now you will be greeted by a new command line prompt that will look something like this
You can now install the software that you need through the image package
manager, in this case
apt-get. If you want
/cvmfs mounted in your image
[root@740b9db736a1 /]# mkdir -p /cvmfs
Once you have installed all the software, you simply
[root@740b9db736a1 /]# exit
Now you can commit the changes to the image
docker commit <docker_session_name_here>
Alternatively, you can use the sessions hash (
740b9db736a1 in the above example).
Then you make a tag for your desired
docker tag opensciencegrid/osgvo-ubuntu-xenial namespace/repository_name
And finally you simply need to push the changes:
docker push namespace/repository_name
Configuring GPU images
GPU images requires a few extra steps to get access to the host GPU libraries.
We recommend that in this case you edit the
Dockerfile rather than building
a image through Docker.
GPU containers running under GPU slots will automatically get the host GPU
libraries bound to
/host-libs inside the container. For this to work,
/host-libs directory has to exist in the image, and
has to be set for the job. An example of how to do this is the
TensorFlow GPU image.
Distributing Custom Images Via CVMFS
In order to be able to efficiently distribute the container images to a large
of distributed compute hosts, OSG has chosen to host the images under
CVMFS. Any image publicly available in
Docker can be included for automatic syncing into the CVMFS repository. The
result is an unpacked image under
Once your image has been registered, new versions pushed to Docker Hub will automatically be detected and CVMFS will be updated accordingly.
Note: some common Dockerfile features, like ENV and ENTRYPOINT, are ignored when the image is imported into CVMFS.
This page was updated on Jun 18, 2019 at 18:00 from start/resources/singularity-containers.md.