Package peppy Documentation

Package Overview

The peppy package provides a Python interface for working with Portable Encapsulated Projects (PEPs). A PEP is a standardized format for organizing metadata for biological samples and sample-intensive data.

Key Features

• Project Management: Create and manage collections of samples with metadata
• Sample Access: Retrieve individual samples and their attributes
• Amendments: Activate different project configurations
• Validation: Validate projects against schemas

Installation

pip install peppy

Quick Example

from peppy import Project

# Initialize with a project config file
prj = Project(cfg="ngs.yaml")

# Access samples
samples = prj.samples

API Reference

Project Class

The main class for working with PEPs:

Project

class Project(cfg: str = None, amendments: Union[str, Iterable[str]] = None, sample_table_index: Union[str, Iterable[str]] = None, subsample_table_index: Union[str, Iterable[str]] = None, defer_samples_creation: bool = False)

Bases: MutableMapping

A class to model a Project (collection of samples and metadata).

:param str cfg: Project config file (YAML) or sample table (CSV/TSV) with one row per sample to constitute project :param str | Iterable[str] sample_table_index: name of the columns to set the sample_table index to :param str | Iterable[str] subsample_table_index: name of the columns to set the subsample_table index to :param str | Iterable[str] amendments: names of the amendments to activate :param Iterable[str] amendments: amendments to use within configuration file :param bool defer_samples_creation: whether the sample creation should be skipped

:Example:

from peppy import Project
prj = Project(cfg="ngs.yaml")
samples = prj.samples

Attributes

• progressbar
• sst_index
• st_index

Properties

amendments

@property
def amendments

Return currently active list of amendments or None if none was activated

:return Iterable[str]: a list of currently active amendment names

config

@property
def config

Get the config mapping

:return Mapping: config. May be formatted to comply with the most recent version specifications

config_file

@property
def config_file

Get the config file path

:return str: path to the config file

description

@property
def description

is_sample_table_large

@property
def is_sample_table_large

list_amendments

@property
def list_amendments

Return a list of available amendments or None if not declared

:return Iterable[str]: a list of available amendment names

name

@property
def name

pep_version

@property
def pep_version

The declared PEP version string

It is validated to make sure it is a valid PEP version string

:raise InvalidConfigFileException: in case of invalid PEP version :return str: PEP version string

sample_name_colname

@property
def sample_name_colname

Deprecated, please use Project.sample_table_index instead

Name of the effective sample name containing column in the sample table.

It is "sample_name" by default, but when it's missing it could be replaced by the selected sample table index, defined on the object instantiation stage.

:return str: name of the column that consist of sample identifiers

sample_table

@property
def sample_table

Get sample table. If any sample edits were performed, it will be re-generated

:return pandas.DataFrame: a data frame with current samples attributes

sample_table_index

@property
def sample_table_index

The effective sample table index.

It is sample_name by default, but could be overwritten by the selected sample table index, defined on the object instantiation stage or in the project configuration file via sample_table_index field.

That's the sample table index selection priority order:

1. Constructor specified
2. Config specified
3. Deafult: sample_table

:return str: name of the column that consist of sample identifiers

samples

@property
def samples

Generic/base Sample instance for each of this Project's samples.

:return Iterable[Sample]: Sample instance for each of this Project's samples

subsample_table

@property
def subsample_table

Get subsample table

:return pandas.DataFrame: a data frame with subsample attributes

subsample_table_index

@property
def subsample_table_index

The effective subsample table indexes.

It is [subasample_name, sample_name] by default, but could be overwritten by the selected subsample table indexes, defined on the object instantiation stage or in the project configuration file via subsample_table_index field.

That's the subsample table indexes selection priority order:

1. Constructor specified
2. Config specified
3. Deafult: [subasample_name, sample_name]

:return List[str]: names of the columns that consist of sample and subsample identifiers

Class methods

from_dict

@classmethod
def from_dict(pep_dictionary: dict)

Init a peppy project instance from a dictionary representation of an already processed PEP.

:param Dict[Any] pep_dictionary: dict representation of the project {_config: dict, _samples: list | dict, _subsamples: list[list | dict]}

from_pandas

@classmethod
def from_pandas(samples_df: pd.DataFrame, sub_samples_df: List[pd.DataFrame] = None, config: dict = None)

Init a peppy project instance from a pandas Dataframe

:param samples_df: in-memory pandas DataFrame object of samples :param sub_samples_df: in-memory list of pandas DataFrame objects of sub-samples :param config: dict of yaml file

from_pep_config

@classmethod
def from_pep_config(cfg: str = None, amendments: Union[str, Iterable[str]] = None, sample_table_index: Union[str, Iterable[str]] = None, subsample_table_index: Union[str, Iterable[str]] = None, defer_samples_creation: bool = False)

Init a peppy project instance from a yaml file

:param str cfg: Project config file (YAML) or sample table (CSV/TSV) with one row per sample to constitute project :param str | Iterable[str] sample_table_index: name of the columns to set the sample_table index to :param str | Iterable[str] subsample_table_index: name of the columns to set the subsample_table index to :param str | Iterable[str] amendments: names of the amendments to activate :param Iterable[str] amendments: amendments to use within configuration file :param bool defer_samples_creation: whether the sample creation should be skipped

from_pephub

@classmethod
def from_pephub(registry_path: str) -> Project

Init project from pephubclient.

:param registry_path: PEPhub registry path :return: peppy Project

from_sample_yaml

@classmethod
def from_sample_yaml(yaml_file: str)

Init a peppy project instance from a yaml file

:param str yaml_file: path to yaml file

Methods

__getitem__

def __getitem__(item)

Fetch the value of given key.

:param hashable item: key for which to fetch value :return object: value mapped to given key, if available :raise KeyError: if the requested key is unmapped.

__str__

def __str__()

Representation in interpreter.

activate_amendments

def activate_amendments(amendments)

Update settings based on amendment-specific values.

This method will update Project attributes, adding new values associated with the amendments indicated, and in case of collision with an existing key/attribute the amendments' values will be favored.

:param Iterable[str] amendments: A string with amendment names to be activated :return peppy.Project: Updated Project instance :raise TypeError: if argument to amendment parameter is null :raise NotImplementedError: if this call is made on a project not created from a config file

add_samples

def add_samples(samples)

Add list of Sample objects

:param peppy.Sample | Iterable[peppy.Sample] samples: samples to add

attr_constants

def attr_constants()

Update each Sample with constants declared by a Project. If Project does not declare constants, no update occurs.

attr_derive

def attr_derive(attrs=None)

Set derived attributes for all Samples tied to this Project instance

attr_imply

def attr_imply()

Infer value for additional field(s) from other field(s).

Add columns/fields to the sample based on values in those already-set that the sample's project defines as indicative of implications for additional data elements for the sample.

attr_merge

def attr_merge()

Merge sample subannotations (from subsample table) with sample annotations (from sample_table)

attr_remove

def attr_remove()

Remove declared attributes from all samples that have them defined

attr_synonyms

def attr_synonyms()

Copy attribute values for all samples to a new one

create_samples

def create_samples(modify: bool = False)

Populate Project with Sample objects

deactivate_amendments

def deactivate_amendments()

Bring the original project settings back.

:return peppy.Project: Updated Project instance :raise NotImplementedError: if this call is made on a project not created from a config file

get_description

def get_description()

Infer project description from config file.

The provided description has to be of class coercible to string

:return str: inferred name for project. :raise InvalidConfigFileException: if description is not of class coercible to string

get_sample

def get_sample(sample_name)

Get an individual sample object from the project.

Will raise a ValueError if the sample is not found. In the case of multiple samples with the same name (which is not typically allowed), a warning is raised and the first sample is returned

:param str sample_name: The name of a sample to retrieve :raise ValueError: if there's no sample with the specified name defined :return peppy.Sample: The requested Sample object

get_samples

def get_samples(sample_names)

Returns a list of sample objects given a list of sample names

:param list sample_names: A list of sample names to retrieve :return list[peppy.Sample]: A list of Sample objects

infer_name

def infer_name()

Infer project name from config file path.

First assume the name is the folder in which the config file resides, unless that folder is named "metadata", in which case the project name is the parent of that folder.

:return str: inferred name for project. :raise InvalidConfigFileException: if the project lacks both a name and a configuration file (no basis, then, for inference) :raise InvalidConfigFileException: if specified Project name is invalid

load_samples

def load_samples()

Read the sample_table and subsample_tables into dataframes and store in the object root. The values sourced from the project config can be overwritten by the optional arguments.

modify_samples

def modify_samples()

Perform any sample modifications defined in the config.

parse_config_file

def parse_config_file(cfg_path: str = None, amendments: Iterable[str] = None)

Parse provided yaml config file and check required fields exist.

:param str cfg_path: path to the config file to read and parse :param Iterable[str] amendments: Name of amendments to activate :raises KeyError: if config file lacks required section(s)

remove_samples

def remove_samples(sample_names)

Remove Samples from Project

:param Iterable[str] sample_names: sample names to remove

to_dict

def to_dict(extended: bool = False, orient: Literal['dict', 'list', 'series', 'split', 'tight', 'records', 'index'] = 'dict') -> dict

Convert the Project object to a dictionary.

:param bool extended: whether to produce complete project dict (used to reinit the project) :param Literal orient: orientation of the returned df :return dict: a dictionary representation of the Project object

Sample Class

Sample

class Sample(series, prj=None)

Bases: SimpleAttMap

Class to model Samples based on a pandas Series.

:param Mapping | pandas.core.series.Series series: Sample's data.

Properties

project

@property
def project

Get the project mapping

:return peppy.Project: project object the sample was created from

sample_name

@property
def sample_name

Get the sample's name

:return str: current sample name derived from project's st_index

Methods

__str__

def __str__(max_attr=10)

Representation in interpreter.

derive_attribute

def derive_attribute(data_sources, attr_name)

Uses the template path provided in the project config section "data_sources" to piece together an actual path by substituting variables (encoded by "{variable}"") with sample attributes.

:param Mapping data_sources: mapping from key name (as a value in a cell of a tabular data structure) to, e.g., filepath :param str attr_name: Name of sample attribute (equivalently, sample sheet column) specifying a derived column. :return str: regex expansion of data source specified in configuration, with variable substitutions made :raises ValueError: if argument to data_sources parameter is null/empty

get_sheet_dict

def get_sheet_dict()

Create a K-V pairs for items originally passed in via the sample sheet. This is useful for summarizing; it provides a representation of the sample that excludes things like config files and derived entries.

:return OrderedDict: mapping from name to value for data elements originally provided via the sample sheet (i.e., the a map-like representation of the instance, excluding derived items)

to_dict

def to_dict(add_prj_ref=False)

Serializes itself as dict object.

:param bool add_prj_ref: whether the project reference bound do the Sample object should be included in the YAML representation :return dict: dict representation of this Sample

to_yaml

def to_yaml(path: Optional[str] = None, add_prj_ref=False) -> Union[str, None]

Serializes itself in YAML format. Writes to file if path is provided, else returns string representation.

:param str path: A file path to write yaml to; provide this or the subs_folder_path, defaults to None :param bool add_prj_ref: whether the project reference bound do the Sample object should be included in the YAML representation :return str | None: returns string representation of sample yaml or None