Looper Python API

The looper Python API provides classes for managing pipeline submissions and compute configurations.

Project

The main class for working with looper projects. Extends peppy's Project with pipeline submission capabilities.

Project

Project(cfg=None, amendments=None, divcfg_path=None, **kwargs)

Bases: Project

Looper-specific Project.

Parameters:

Name	Type	Description	Default
cfg	str	Path to configuration file with data from which Project is to be built.	None
amendments	Iterable[str]	Name indicating amendment to use, optional.	None
divcfg_path	str	Path to an environment configuration YAML file specifying compute settings.	None

cli_pifaces property

cli_pifaces

Collection of pipeline interface sources specified in object constructor.

Returns:

Type	Description
list[str] | None	list[str]: Collection of pipeline interface sources.

output_dir property

output_dir

Output directory for the project, specified in object constructor.

Returns:

Name	Type	Description
str	str	Path to the output directory.

pipeline_interfaces cached property

pipeline_interfaces

Flat list of all valid interface objects associated with this Project.

Note that only valid pipeline interfaces will show up in the result (ones that exist on disk/remotely and validate successfully against the schema).

Returns:

Type	Description
list	list[looper.PipelineInterface]: List of pipeline interfaces.

results_folder property

results_folder

Path to the results folder for the project.

Returns:

Name	Type	Description
str	str	Path to the results folder in the output folder.

selected_compute_package property

selected_compute_package

Compute package name specified in object constructor.

Returns:

Name	Type	Description
str	str | None	Compute package name.

submission_folder property

submission_folder

Path to the submission folder for the project.

Returns:

Name	Type	Description
str	str	Path to the submission in the output folder.

get_sample_piface

get_sample_piface(sample_name)

Get a list of pipeline interfaces associated with the specified sample.

Note that only valid pipeline interfaces will show up in the result (ones that exist on disk/remotely and validate successfully against the schema).

Parameters:

Name	Type	Description	Default
sample_name	str	Name of the sample to retrieve list of pipeline interfaces for.	required

Returns:

Type	Description
list | None	list[looper.PipelineInterface]: Collection of valid pipeline interfaces associated with selected sample.

get_schemas staticmethod

get_schemas(pifaces, schema_key=INPUT_SCHEMA_KEY)

Get the list of unique schema paths for a list of pipeline interfaces.

Parameters:

Name	Type	Description	Default
pifaces	str | Iterable[str]	Pipeline interfaces to search schemas for.	required
schema_key	str	Where to look for schemas in the piface.	INPUT_SCHEMA_KEY

Returns:

Type	Description
list[str]	Iterable[str]: Unique list of schema file paths.

make_project_dirs

make_project_dirs()

Create project directory structure if it doesn't exist.

populate_pipeline_outputs

populate_pipeline_outputs()

Populate project and sample output attributes based on output schemas that pipeline interfaces point to.

set_sample_piface

set_sample_piface(sample_piface)

Add sample pipeline interfaces variable to object.

Parameters:

Name	Type	Description	Default
sample_piface	list | str	Sample pipeline interface.	required

PipelineInterface

Parses and holds information from a pipeline interface YAML file, including resource specifications and command templates.

PipelineInterface

PipelineInterface(config, pipeline_type=None)

Bases: YAMLConfigManager

This class parses, holds, and returns information for a yaml file that specifies how to interact with each individual pipeline.

This includes both resources to request for cluster job submission, as well as arguments to be passed from the sample annotation metadata to the pipeline.

Parameters:

Name	Type	Description	Default
config	str | Mapping	Path to file from which to parse configuration data, or pre-parsed configuration data.	required
pipeline_type	str	Type of the pipeline, must be either 'sample' or 'project'.	None

pipeline_name property

pipeline_name

choose_resource_package

choose_resource_package(namespaces, file_size)

Select resource bundle for given input file size to given pipeline.

Parameters:

Name	Type	Description	Default
file_size	float	Size of input data (in gigabytes).	required
namespaces	Mapping[Mapping[str]]	Namespaced variables to pass as a context for fluid attributes command rendering.	required

Returns:

Name	Type	Description
MutableMapping	dict	Resource bundle appropriate for given pipeline, for given input file size.

Raises:

Type	Description
ValueError	If indicated file size is negative, or if the file size value specified for any resource package is negative.
InvalidResourceSpecificationException	If no default resource package specification is provided.

get_pipeline_schemas

get_pipeline_schemas(schema_key=INPUT_SCHEMA_KEY)

Get path to the pipeline schema.

Parameters:

Name	Type	Description	Default
schema_key	str	Where to look for schemas in the pipeline iface.	INPUT_SCHEMA_KEY

Returns:

Name	Type	Description
str	str | None	Absolute path to the pipeline schema file.

render_var_templates

render_var_templates(namespaces)

Render path templates under 'var_templates' in this pipeline interface.

Parameters:

Name	Type	Description	Default
namespaces	dict	Namespaces to use for rendering.	required

SubmissionConductor

Collects and submits pipeline jobs. Manages job pooling based on file size or command count limits.

SubmissionConductor

SubmissionConductor(pipeline_interface, prj, delay=0, extra_args=None, extra_args_override=None, ignore_flags=False, compute_variables=None, max_cmds=None, max_size=None, max_jobs=None, automatic=True, collate=False)

Bases: object

Collects and then submits pipeline jobs.

This class holds a 'pool' of commands to submit as a single cluster job. Eager to submit a job, each instance's collection of commands expands until it reaches the 'pool' has been filled, and it's therefore time to submit the job. The pool fills as soon as a fill criteria has been reached, which can be either total input file size or the number of individual commands.

Create a job submission manager.

The most critical inputs are the pipeline interface and the pipeline key, which together determine which provide critical pipeline information like resource allocation packages and which pipeline will be overseen by this instance, respectively.

Parameters:

Name	Type	Description	Default
pipeline_interface	PipelineInterface	Collection of important data for one or more pipelines, like resource allocation packages and option/argument specifications.	required
prj	Project	Project with which each sample being considered is associated (what generated each sample).	required
delay	float	Time (in seconds) to wait before submitting a job once it's ready.	0
extra_args	str	String to pass to each job generated, for example additional pipeline arguments.	None
extra_args_override	str	String to pass to each job generated, for example additional pipeline arguments. This deactivates the 'extra' functionality that appends strings defined in Sample.command_extra and Project.looper.command_extra to the command template.	None
ignore_flags	bool	Whether to ignore flag files present in the sample folder for each sample considered for submission.	False
compute_variables	dict[str]	A dict with variables that will be made available to the compute package. For example, this should include the name of the cluster partition to which job or jobs will be submitted.	None
max_cmds	int | None	Upper bound on number of commands to include in a single job script.	None
max_size	int | float | None	Upper bound on total file size of inputs used by the commands lumped into single job script.	None
max_jobs	int | float | None	Upper bound on total number of jobs to group samples for submission.	None
automatic	bool	Whether the submission should be automatic once the pool reaches capacity.	True
collate	bool	Whether a collate job is to be submitted (runs on the project level, rather that on the sample level).	False

failed_samples property

failed_samples

num_cmd_submissions property

num_cmd_submissions

Return the number of commands that this conductor has submitted.

Returns:

Name	Type	Description
int	int	Number of commands submitted so far.

num_job_submissions property

num_job_submissions

Return the number of jobs that this conductor has submitted.

Returns:

Name	Type	Description
int	int	Number of jobs submitted so far.

add_sample

add_sample(sample, rerun=False)

Add a sample for submission to this conductor.

Parameters:

Name	Type	Description	Default
sample	Sample	Sample to be included with this conductor's currently growing collection of command submissions.	required
rerun	bool	Whether the given sample is being rerun rather than run for the first time.	False

Returns:

Name	Type	Description
list	list	List of skip reasons if sample was not added.

Raises:

Type	Description
TypeError	If sample subtype is provided but does not extend the base Sample class.

is_project_submittable

is_project_submittable(force=False)

Check whether the current project has been already submitted.

Parameters:

Name	Type	Description	Default
force	bool	Whether to force the project submission (ignore status/flags).	False

Returns:

Name	Type	Description
bool	bool	True if the project is submittable, False otherwise.

submit

submit(force=False)

Submit one or more commands as a job.

This call will submit the commands corresponding to the current pool of samples if and only if the argument to 'force' evaluates to a true value, or the pool of samples is full.

Parameters:

Name	Type	Description	Default
force	bool	Whether submission should be done/simulated even if this conductor's pool isn't full.	False

Returns:

Name	Type	Description
bool	bool	Whether a job was submitted (or would've been if not for dry run).

write_script

write_script(pool, size)

Create the script for job submission.

Parameters:

Name	Type	Description	Default
pool	Iterable[Sample]	Collection of sample instances.	required
size	float	Cumulative size of the given pool.	required

Returns:

Name	Type	Description
str	str	Path to the job submission script created.

ComputingConfiguration

Manages compute environment settings from divvy configuration files. Handles resource packages and submission templates.

ComputingConfiguration

ComputingConfiguration(entries=None, wait_max=None, strict_ro_locks=False)

Bases: YAMLConfigManager

Represents computing configuration objects.

The ComputingConfiguration class provides a computing configuration object that is an in memory representation of a divvy computing configuration file. This object has various functions to allow a user to activate, modify, and retrieve computing configuration files, and use these values to populate job submission script templates.

Parameters:

Name	Type	Description	Default
entries	str | Iterable[str, object] | Mapping[str, object]	Config collection of key-value pairs.	None

default_config_file property

default_config_file

Path to default compute environment settings file.

Returns:

Name	Type	Description
str	str	Path to default compute settings file.

templates_folder property

templates_folder

Path to folder with default submission templates.

Returns:

Name	Type	Description
str	str	Path to folder with default submission templates.

activate_package

activate_package(package_name)

Activates a compute package.

This copies the computing attributes from the configuration file into the compute attribute, where the class stores current compute settings.

Parameters:

Name	Type	Description	Default
package_name	str	Name for non-resource compute bundle, the name of a subsection in an environment configuration file.	required

Returns:

Name	Type	Description
bool	bool	Success flag for attempt to establish compute settings.

clean_start

clean_start(package_name)

Clear current active settings and then activate the given package.

Parameters:

Name	Type	Description	Default
package_name	str	Name of the resource package to activate.	required

Returns:

Name	Type	Description
bool	bool	Success flag.

get_active_package

get_active_package()

Returns settings for the currently active compute package.

Returns:

Name	Type	Description
YAMLConfigManager	YAMLConfigManager	Data defining the active compute package.

get_adapters

get_adapters()

Get current adapters, if defined.

Adapters are sourced from the 'adapters' section in the root of the divvy configuration file and updated with an active compute package-specific set of adapters, if any defined in 'adapters' section under currently active compute package.

Returns:

Name	Type	Description
YAMLConfigManager	YAMLConfigManager	Current adapters mapping.

list_compute_packages

list_compute_packages()

Returns a list of available compute packages.

Returns:

Type	Description
set[str]	set[str]: Names of available compute packages.

reset_active_settings

reset_active_settings()

Clear out current compute settings.

Returns:

Name	Type	Description
bool	bool	Success flag.

template

template()

Get the currently active submission template.

Returns:

Name	Type	Description
str	str	Submission script content template for current state.

update_packages

update_packages(config_file)

Parse data from divvy configuration file.

Given a divvy configuration file, this function will update (not overwrite) existing compute packages with existing values. It does not affect any currently active settings.

Parameters:

Name	Type	Description	Default
config_file	str	Path to file with new divvy configuration data.	required

write_script

write_script(output_path, extra_vars=None)

Given currently active settings, populate the active template to write a submission script.

Additionally use the current adapters to adjust the select of the provided variables.

Parameters:

Name	Type	Description	Default
output_path	str	Path to file to write as submission script.	required
extra_vars	Iterable[Mapping]	A list of Dict objects with key-value pairs with which to populate template fields. These will override any values in the currently active compute package.	None

Returns:

Name	Type	Description
str	str	Path to the submission script file.

Utility Functions

select_divvy_config

select_divvy_config

select_divvy_config(filepath)

Selects the divvy config file path to load.

This uses a priority ordering to first choose a config file path if it's given, but if not, then look in a priority list of environment variables and choose the first available file path to return. If none of these options succeed, the default config path will be returned.

Parameters:

Name	Type	Description	Default
filepath	str | NoneType	Direct file path specification.	required

Returns:

Name	Type	Description
str	str	Path to the config file to read.