Skip to content

Learn amendments in peppy

This vignette will show you how and why to use the amendments functionality of the peppy package.

Problem/Goal

The example below demonstrates how and why to use amendments project attribute to, e.g. define numerous similar projects in a single project config file. This functionality is extremely convenient when one has to define projects with small settings discreptancies, like different attributes in the annotation sheet. For example libraries ABCD and EFGH instead of the original RRBS.

examples_dir = "../tests/data/example_peps-cfg2/example_amendments1/"
sample_table = examples_dir + "sample_table.csv"
%cat $sample_table | column -t -s, | cat

sample_name  protocol  organism  time  file_path

pig_0h       RRBS      pig       0     source1

pig_1h       RRBS      pig       1     source1

frog_0h      RRBS      frog      0     source1

frog_1h      RRBS      frog      1     source1

Solution

This can be achieved by using amendments section of project_config.yaml file (presented below). The attributes specified in the lowest levels of this section (here: sample_table) overwrite the original ones. Consequently, a completely new set of settings is determined with just this value changed. Moreover, multiple amendments can be defined in a single config file and activated at the same time. Based on the file presented below, two subprojects will be defined: newLib and newLib2.

project_config_file = examples_dir + "project_config.yaml"
%cat $project_config_file

pep_version: "2.0.0"
sample_table: sample_table.csv
output_dir: $HOME/hello_looper_results

sample_modifiers:
  derive:
    attributes: [file_path]
    sources:
      source1: /data/lab/project/{organism}_{time}h.fastq
      source2: /path/from/collaborator/weirdNamingScheme_{external_id}.fastq
project_modifiers:
  amend:
    newLib:
      sample_table: sample_table_newLib.csv
    newLib2:
      sample_table: sample_table_newLib2.csv

Obviously, the amendments functionality can be combined with other peppy package options, e.g. imply and derive sample modifiers. The derive modifier is used in the example considered here (derive key in the sample_modifiers section of the config file).

Files sample_table_newLib.csv and sample_table_newLib2.csv introduce different the library attributes. They are used in the subprojects newLib and newLib2, respectively

sample_table = examples_dir + "sample_table_newLib.csv"
%cat $sample_table | column -t -s, | cat

sample_name  protocol  organism  time  file_path
pig_0h       ABCD      pig       0     source1
pig_1h       ABCD      pig       1     source1
frog_0h      ABCD      frog      0     source1
frog_1h      ABCD      frog      1     source1

sample_table = examples_dir + "sample_table_newLib2.csv"
%cat $sample_table | column -t -s, | cat

sample_name  protocol  organism  time  file_path
pig_0h       EFGH      pig       0     source1
pig_1h       EFGH      pig       1     source1
frog_0h      EFGH      frog      0     source1
frog_1h      EFGH      frog      1     source1

Code

Import peppy and read in the project metadata by specifying the path to the project_config.yaml

from peppy import Project
p = Project(project_config_file)

An appropriate message is displayed, which informs you what are the names of the amendments that you have defined in the project_config.yaml file. Nonetheless, just the main project is "active".

Let's inspect it:

p.sample_table
file_path organism protocol sample_name time
sample_name
pig_0h /data/lab/project/pig_0h.fastq pig RRBS pig_0h 0
pig_1h /data/lab/project/pig_1h.fastq pig RRBS pig_1h 1
frog_0h /data/lab/project/frog_0h.fastq frog RRBS frog_0h 0
frog_1h /data/lab/project/frog_1h.fastq frog RRBS frog_1h 1

The column file_path was derived and the library column holds the original attributes: RRBS for each sample.

To "activate" any of the amendments just pass the names of the desired amendments to the amendments argument in the Project object constructor.

In case you don't remember the subproject names run the listAmendments() metohods on the Project object, just like that:

p.list_amendments

['newLib', 'newLib2']

p_new_lib = Project(project_config_file, amendments = "newLib")

Let's inspect it:

p_new_lib.sample_table
file_path organism protocol sample_name time
sample_name
pig_0h /data/lab/project/pig_0h.fastq pig ABCD pig_0h 0
pig_1h /data/lab/project/pig_1h.fastq pig ABCD pig_1h 1
frog_0h /data/lab/project/frog_0h.fastq frog ABCD frog_0h 0
frog_1h /data/lab/project/frog_1h.fastq frog ABCD frog_1h 1

As you can see, the library column consists of new attributes (ABCD), which were defined in the sample_table_newLib.csv file.

Amendments can be also activated interactively, after Project object has been crated. Let's activate the second amendment this way:

p_new_lib2 = p.activate_amendments("newLib2")
p_new_lib2.sample_table
file_path organism protocol sample_name time
sample_name
pig_0h /data/lab/project/pig_0h.fastq pig EFGH pig_0h 0
pig_1h /data/lab/project/pig_1h.fastq pig EFGH pig_1h 1
frog_0h /data/lab/project/frog_0h.fastq frog EFGH frog_0h 0
frog_1h /data/lab/project/frog_1h.fastq frog EFGH frog_1h 1