How to run jobs in containers

Looper supports running pipelines inside containers. The recommended approach is to use bulker, which manages lightweight, modular, single-tool containers. For pipelines that require a monolithic container (one large image containing the entire pipeline), looper also provides built-in Docker and Apptainer compute packages, described below.

1. Get your container image.

This could be a docker image (hosted on dockerhub), which you would download via docker pull, or it could be an apptainer image you have saved in a local folder. This is pipeline-specific, and you'll need to download the image recommended by the authors of the pipeline or pipelines you want to run.

2. Specify the image in your `pipeline_interface`

The pipeline_interface.yaml file will need a compute section for each pipeline that can be run in a container, specifying the image. For example:

compute:
  apptainer_image: ${SIMAGES}myimage
  docker_image: databio/myimage

For apptainer images, you just need to make sure that the images indicated in the pipeline_interface are available in those locations on your system. For docker, make sure you have the docker images pulled.

3. Run with a built-in container package

Looper ships with built-in compute packages for Apptainer and Docker, so you don't need to create your own. Just select the appropriate package:

# Apptainer locally
looper run --package apptainer

# Apptainer on SLURM
looper run --package apptainer_slurm \
  --compute partition=standard time='04:00:00' cores='8' mem='16000'

# Docker locally
looper run --package docker

To pass extra arguments to the container runtime (e.g., bind mounts for apptainer), use --compute:

looper run --package apptainer_slurm \
  --compute partition=standard time='04:00:00' cores='8' mem='16000' \
  apptainer_args='-B /sfs/lustre:/sfs/lustre,/nm/t1:/nm/t1'

Or set these in your looper config to avoid typing them every time:

cli:
  run:
    package: apptainer_slurm
    compute:
      partition: standard
      time: '04:00:00'
      cores: '8'
      mem: '16000'
      apptainer_args: -B /sfs/lustre:/sfs/lustre,/nm/t1:/nm/t1

Available container packages

Package	Runtime	Submission	Description
`apptainer`	Apptainer	`sh` (local)	Run in an apptainer container locally
`apptainer_slurm`	Apptainer	`sbatch`	Submit apptainer jobs to SLURM
`docker`	Docker	`sh` (local)	Run in a docker container locally