How to create custom submission templates

Looper uses divvy compute packages to define how jobs are submitted. Each compute package pairs a submission template with a submission command. You can create your own packages to support any computing scenario.

Steps

1. Create a submission template

Create a text file that serves as the skeleton for your job script. Use {VARIABLE} syntax for any values that should be filled in at run time. At minimum, include {CODE}, which looper populates with the pipeline command.

Common template variables:

{CODE} -- the pipeline command (populated by looper)
{JOBNAME} -- job name (populated by looper)
{LOGFILE} -- log file path (populated by looper)
{MEM}, {CORES}, {TIME}, {PARTITION} -- resource requests
{DOCKER_IMAGE}, {APPTAINER_IMAGE}, {APPTAINER_ARGS}, {DOCKER_ARGS} -- container settings
{PRE_COMMAND}, {POST_COMMAND} -- command hooks for pre/post execution steps
{BULKER_CRATE} -- bulker crate identifier

You can also introduce your own custom variables (see the SLURM with custom account example below).

2. Add a compute package to your divvy config

Open your divvy config file and add a new entry under compute_packages.

3. Set the `submission_template` to point to your template file

The path can be absolute or relative to the divvy config file.

4. Set the `submission_command` for this package

This is the command divvy uses to submit the generated script (e.g. sh, sbatch, qsub).

Example: SLURM with a custom account variable

Suppose your cluster requires an --account flag on every job. Create a template that includes a custom {ACCOUNT} variable:

#!/bin/bash
#SBATCH --job-name='{JOBNAME}'
#SBATCH --output='{LOGFILE}'
#SBATCH --mem='{MEM}'
#SBATCH --cpus-per-task='{CORES}'
#SBATCH --time='{TIME}'
#SBATCH --partition='{PARTITION}'
#SBATCH --account='{ACCOUNT}'
#SBATCH --ntasks=1

echo 'Compute node:' `hostname`
echo 'Start time:' `date +'%Y-%m-%d %T'`

{PRE_COMMAND}

srun {CODE}

{POST_COMMAND}

Then register it in your divvy config, setting the account value as a package-level default:

compute_packages:
  default:
    submission_template: divvy_templates/local_template.sub
    submission_command: sh
  slurm_account:
    submission_template: divvy_templates/slurm_account_template.sub
    submission_command: sbatch
    account: my_lab_allocation

Run with:

looper run --package slurm_account

You can override the account (or any other variable) at run time:

looper run --package slurm_account --compute account=other_allocation