Skip to content
PEPkit

eido command line usage

To use the command line application one just needs a path to a project configuration file. It is a positional argument in the eido command.

For this tutorial, let's grab a PEP from a public example repository that describes a few PRO-seq test samples:

Terminal window
rm -rf ppqc
git clone https://github.com/databio/ppqc.git --branch cfg2
Cloning into 'ppqc'...
remote: Enumerating objects: 154, done.
remote: Counting objects: 100% (20/20), done.
remote: Compressing objects: 100% (15/15), done.
remote: Total 154 (delta 7), reused 17 (delta 5), pack-reused 134
Receiving objects: 100% (154/154), 81.69 KiB | 3.27 MiB/s, done.
Resolving deltas: 100% (82/82), done.
Terminal window
cd ppqc
export DATA=$HOME
export SRAFQ=$HOME

PEP inspection

Section titled “PEP inspection”

First, let's use eido inspect to inspect a PEP.

  • To inspect the entire Project object just provide the path to the project configuration file.
Terminal window
eido inspect peppro_paper.yaml
Project 'PEPPRO' (peppro_paper.yaml)
47 samples (showing first 20): K562_PRO-seq_02, K562_PRO-seq_04, K562_PRO-seq_06, K562_PRO-seq_08, K562_PRO-seq_10, K562_PRO-seq_20, K562_PRO-seq_30, K562_PRO-seq_40, K562_PRO-seq_50, K562_PRO-seq_60, K562_PRO-seq_70, K562_PRO-seq_80, K562_PRO-seq_90, K562_PRO-seq_100, K562_RNA-seq_0, K562_RNA-seq_10, K562_RNA-seq_20, K562_RNA-seq_30, K562_RNA-seq_40, K562_RNA-seq_50
Sections: name, pep_version, sample_table, looper, sample_modifiers
  • To inspect a specific sample, one needs to provide the sample name (via -n/--sample-name oprional argument)
Terminal window
eido inspect peppro_paper.yaml -n K562_PRO-seq K562_RNA-seq_10
Sample 'K562_RNA-seq_10' in Project (peppro_paper.yaml)


sample_name:         K562_RNA-seq_10
sample_desc:         90% K562 PRO-seq + 10% K562 RNA-seq
treatment:           70M total reads
protocol:            PRO
organism:            human
read_type:           SINGLE
umi_len:             0
read1:               /Users/mstolarczyk/K562_10pctRNA.fastq.gz
srr:                 K562_10pctRNA
pipeline_interfaces: $CODE/peppro/sample_pipeline_interface.yaml
genome:              hg38


...                (showing first 10)

PEP validation

Section titled “PEP validation”

Next, let's use eido to validate this project against the generic PEP schema. You just need to provide a path to the project config file and schema as an input.

Terminal window
eido validate peppro_paper.yaml -s http://schema.databio.org/pep/2.0.0.yaml -e
Validation successful

Any PEP should validate against that schema, which describes generic PEP format. We can go one step further and validate it against the PEPPRO schema, which describes Proseq projects specfically for this pipeline:

Terminal window
eido validate peppro_paper.yaml -s http://schema.databio.org/pipelines/ProseqPEP.yaml
Validation successful

This project would not validate against a different pipeline's schema.

Following jsonschema, eido produces comprehensive error messages that include the objects that did not pass validation. When validating PEPs that include lots of samples one can use option -e/--exclude-case to limit the error output just to the human readable message. This is the option used in the example below:

Terminal window
eido validate peppro_paper.yaml -s http://schema.databio.org/pipelines/bedmaker.yaml -e
Traceback (most recent call last):
  File "/usr/local/bin/eido", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.9/site-packages/eido/cli.py", line 89, in main
    validate_project(p, args.schema, args.exclude_case)
  File "/usr/local/lib/python3.9/site-packages/eido/validation.py", line 45, in validate_project
    _validate_object(project_dict, preprocess_schema(schema_dict), exclude_case)
  File "/usr/local/lib/python3.9/site-packages/eido/validation.py", line 30, in _validate_object
    raise jsonschema.exceptions.ValidationError(e.message)
jsonschema.exceptions.ValidationError: 'input_file_path' is a required property

Optionally, to validate just the config part of the PEP or a specific sample, -n/--sample-name or -c/--just-config arguments should be used, respectively. Please refer to the help for more details:

Terminal window
eido validate -h
usage: eido validate [-h] -s S [-e] [-n S | -c] PEP


Validate the PEP or its components.


positional arguments:
  PEP                   Path to a PEP configuration file in yaml format.


optional arguments:
  -h, --help            show this help message and exit
  -s S, --schema S      Path to a PEP schema file in yaml format.
  -e, --exclude-case    Whether to exclude the validation case from an error.
                        Only the human readable message explaining the error
                        will be raised. Useful when validating large PEPs.
  -n S, --sample-name S
                        Name or index of the sample to validate. Only this
                        sample will be validated.
  -c, --just-config     Whether samples should be excluded from the
                        validation.

PEP conversion

Section titled “PEP conversion”

Let's use eido convert command to convert PEPs to a variety of different formats. eido supports a plugin system, which can be used by other tool developers to create Python plugin functions that save PEPs in a desired format. Please refer to the documentation for more details. For now let's focus on a couple of plugins that are built-in in eido.

To see what plugins are currently avaialable in your Python environment call:

Terminal window
eido filters
Available filters:
 - basic
 - csv
 - yaml
 - yaml-samples
Terminal window
eido convert peppro_paper.yaml --format basic
Running plugin basic
Project 'PEPPRO' (peppro_paper.yaml)
47 samples (showing first 20): K562_PRO-seq_02, K562_PRO-seq_04, K562_PRO-seq_06, K562_PRO-seq_08, K562_PRO-seq_10, K562_PRO-seq_20, K562_PRO-seq_30, K562_PRO-seq_40, K562_PRO-seq_50, K562_PRO-seq_60, K562_PRO-seq_70, K562_PRO-seq_80, K562_PRO-seq_90, K562_PRO-seq_100, K562_RNA-seq_0, K562_RNA-seq_10, K562_RNA-seq_20, K562_RNA-seq_30, K562_RNA-seq_40, K562_RNA-seq_50
Sections: name, pep_version, sample_table, looper, sample_modifiers
Terminal window
eido convert peppro_paper.yaml --format csv
Running plugin csv
sample_name,genome,organism,pipeline_interfaces,prealignments,protocol,read1,read_type,sample_desc,sample_name,srr,treatment,umi_len,read2
K562_PRO-seq_02,hg38,human,['$CODE/peppro/sample_pipeline_interface.yaml'],human_rDNA,PRO,/Users/mstolarczyk/K562_PRO_2pct.fastq.gz,SINGLE,2% subsample of K562 PRO-seq,K562_PRO-seq_02,K562_PRO_2pct,2% subsample,0,
K562_PRO-seq_04,hg38,human,['$CODE/peppro/sample_pipeline_interface.yaml'],human_rDNA,PRO,/Users/mstolarczyk/K562_PRO_4pct.fastq.gz,SINGLE,4% subsample of K562 PRO-seq,K562_PRO-seq_04,K562_PRO_4pct,4% subsample,0,
K562_PRO-seq_06,hg38,human,['$CODE/peppro/sample_pipeline_interface.yaml'],human_rDNA,PRO,/Users/mstolarczyk/K562_PRO_6pct.fastq.gz,SINGLE,6% subsample of K562 PRO-seq,K562_PRO-seq_06,K562_PRO_6pct,6% subsample,0,
K562_PRO-seq_08,hg38,human,['$CODE/peppro/sample_pipeline_interface.yaml'],human_rDNA,PRO,/Users/mstolarczyk/K562_PRO_8pct.fastq.gz,SINGLE,8% subsample of K562 PRO-seq,K562_PRO-seq_08,K562_PRO_8pct,8% subsample,0,
K562_PRO-seq_10,hg38,human,['$CODE/peppro/sample_pipeline_interface.yaml'],human_rDNA,PRO,/Users/mstolarczyk/K562_PRO_10pct.fastq.gz,SINGLE,10% subsample of K562 PRO-seq,K562_PRO-seq_10,K562_PRO_10pct,10% subsample,0,
K562_PRO-seq_20,hg38,human,['$CODE/peppro/sample_pipeline_interface.yaml'],human_rDNA,PRO,/Users/mstolarczyk/K562_PRO_20pct.fastq.gz,SINGLE,20% subsample of K562 PRO-seq,K562_PRO-seq_20,K562_PRO_20pct,20% subsample,0,
K562_PRO-seq_30,hg38,human,['$CODE/peppro/sample_pipeline_interface.yaml'],human_rDNA,PRO,/Users/mstolarczyk/K562_PRO_30pct.fastq.gz,SINGLE,30% subsample of K562 PRO-seq,K562_PRO-seq_30,K562_PRO_30pct,30% subsample,0,
K562_PRO-seq_40,hg38,human,['$CODE/peppro/sample_pipeline_interface.yaml'],human_rDNA,PRO,/Users/mstolarczyk/K562_PRO_40pct.fastq.gz,SINGLE,40% subsample of K562 PRO-seq,K562_PRO-seq_40,K562_PRO_40pct,40% subsample,0,
K562_PRO-seq_50,hg38,human,['$CODE/peppro/sample_pipeline_interface.yaml'],human_rDNA,PRO,/Users/mstolarczyk/K562_PRO_50pct.fastq.gz,SINGLE,50% subsample of K562 PRO-seq,K562_PRO-seq_50,K562_PRO_50pct,50% subsample,0,
K562_PRO-seq_60,hg38,human,['$CODE/peppro/sample_pipeline_interface.yaml'],human_rDNA,PRO,/Users/mstolarczyk/K562_PRO_60pct.fastq.gz,SINGLE,60% subsample of K562 PRO-seq,K562_PRO-seq_60,K562_PRO_60pct,60% subsample,0,
K562_PRO-seq_70,hg38,human,['$CODE/peppro/sample_pipeline_interface.yaml'],human_rDNA,PRO,/Users/mstolarczyk/K562_PRO_70pct.fastq.gz,SINGLE,70% subsample of K562 PRO-seq,K562_PRO-seq_70,K562_PRO_70pct,70% subsample,0,
K562_PRO-seq_80,hg38,human,['$CODE/peppro/sample_pipeline_interface.yaml'],human_rDNA,PRO,/Users/mstolarczyk/K562_PRO_80pct.fastq.gz,SINGLE,80% subsample of K562 PRO-seq,K562_PRO-seq_80,K562_PRO_80pct,80% subsample,0,
K562_PRO-seq_90,hg38,human,['$CODE/peppro/sample_pipeline_interface.yaml'],human_rDNA,PRO,/Users/mstolarczyk/K562_PRO_90pct.fastq.gz,SINGLE,90% subsample of K562 PRO-seq,K562_PRO-seq_90,K562_PRO_90pct,90% subsample,0,
K562_PRO-seq_100,hg38,human,['$CODE/peppro/sample_pipeline_interface.yaml'],human_rDNA,PRO,/Users/mstolarczyk/SRR155431[1-2].fastq.gz,SINGLE,Unsampled K562 PRO-seq,K562_PRO-seq_100,SRR155431[1-2],none,0,
K562_RNA-seq_0,hg38,human,['$CODE/peppro/sample_pipeline_interface.yaml'],human_rDNA,PRO,/Users/mstolarczyk/K562_0pctRNA.fastq.gz,SINGLE,100% K562 PRO-seq + 0% K562 RNA-seq,K562_RNA-seq_0,K562_0pctRNA,70M total reads,0,
K562_RNA-seq_10,hg38,human,['$CODE/peppro/sample_pipeline_interface.yaml'],human_rDNA,PRO,/Users/mstolarczyk/K562_10pctRNA.fastq.gz,SINGLE,90% K562 PRO-seq + 10% K562 RNA-seq,K562_RNA-seq_10,K562_10pctRNA,70M total reads,0,
K562_RNA-seq_20,hg38,human,['$CODE/peppro/sample_pipeline_interface.yaml'],human_rDNA,PRO,/Users/mstolarczyk/K562_20pctRNA.fastq.gz,SINGLE,80% K562 PRO-seq + 20% K562 RNA-seq,K562_RNA-seq_20,K562_20pctRNA,70M total reads,0,
K562_RNA-seq_30,hg38,human,['$CODE/peppro/sample_pipeline_interface.yaml'],human_rDNA,PRO,/Users/mstolarczyk/K562_30pctRNA.fastq.gz,SINGLE,70% K562 PRO-seq + 30% K562 RNA-seq,K562_RNA-seq_30,K562_30pctRNA,70M total reads,0,
K562_RNA-seq_40,hg38,human,['$CODE/peppro/sample_pipeline_interface.yaml'],human_rDNA,PRO,/Users/mstolarczyk/K562_40pctRNA.fastq.gz,SINGLE,60% K562 PRO-seq + 40% K562 RNA-seq,K562_RNA-seq_40,K562_40pctRNA,70M total reads,0,
K562_RNA-seq_50,hg38,human,['$CODE/peppro/sample_pipeline_interface.yaml'],human_rDNA,PRO,/Users/mstolarczyk/K562_50pctRNA.fastq.gz,SINGLE,50% K562 PRO-seq + 50% K562 RNA-seq,K562_RNA-seq_50,K562_50pctRNA,70M total reads,0,
K562_RNA-seq_60,hg38,human,['$CODE/peppro/sample_pipeline_interface.yaml'],human_rDNA,PRO,/Users/mstolarczyk/K562_60pctRNA.fastq.gz,SINGLE,40% K562 PRO-seq + 60% K562 RNA-seq,K562_RNA-seq_60,K562_60pctRNA,70M total reads,0,
K562_RNA-seq_70,hg38,human,['$CODE/peppro/sample_pipeline_interface.yaml'],human_rDNA,PRO,/Users/mstolarczyk/K562_70pctRNA.fastq.gz,SINGLE,30% K562 PRO-seq + 70% K562 RNA-seq,K562_RNA-seq_70,K562_70pctRNA,70M total reads,0,
K562_RNA-seq_80,hg38,human,['$CODE/peppro/sample_pipeline_interface.yaml'],human_rDNA,PRO,/Users/mstolarczyk/K562_80pctRNA.fastq.gz,SINGLE,20% K562 PRO-seq + 80% K562 RNA-seq,K562_RNA-seq_80,K562_80pctRNA,70M total reads,0,
K562_RNA-seq_90,hg38,human,['$CODE/peppro/sample_pipeline_interface.yaml'],human_rDNA,PRO,/Users/mstolarczyk/K562_90pctRNA.fastq.gz,SINGLE,10% K562 PRO-seq + 90% K562 RNA-seq,K562_RNA-seq_90,K562_90pctRNA,70M total reads,0,
K562_RNA-seq_100,hg38,human,['$CODE/peppro/sample_pipeline_interface.yaml'],human_rDNA,PRO,/Users/mstolarczyk/K562_100pctRNA.fastq.gz,SINGLE,0% K562 PRO-seq + 100% K562 RNA-seq,K562_RNA-seq_100,K562_100pctRNA,70M total reads,0,
K562_GRO-seq,hg38,human,['$CODE/peppro/sample_pipeline_interface.yaml'],human_rDNA,GRO,/Users/mstolarczyk/SRR1552484.fastq.gz,SINGLE,K562 GRO-seq,K562_GRO-seq,SRR1552484,none,0,
HelaS3_GRO-seq,hg38,human,['$CODE/peppro/sample_pipeline_interface.yaml'],human_rDNA,GRO,/Users/mstolarczyk/SRR169361[1-2].fastq.gz,SINGLE,HelaS3 GRO-seq,HelaS3_GRO-seq,SRR169361[1-2],none,0,
Jurkat_ChRO-seq_1,hg38,human,['$CODE/peppro/sample_pipeline_interface.yaml'],human_rDNA,PRO,/Users/mstolarczyk/SRR7616133.fastq.gz,SINGLE,Jurkat ChRO-seq,Jurkat_ChRO-seq_1,SRR7616133,none,6,
Jurkat_ChRO-seq_2,hg38,human,['$CODE/peppro/sample_pipeline_interface.yaml'],human_rDNA,PRO,/Users/mstolarczyk/SRR7616134.fastq.gz,SINGLE,Jurkat ChRO-seq,Jurkat_ChRO-seq_2,SRR7616134,none,6,
HEK_PRO-seq,hg38,human,['$CODE/peppro/sample_pipeline_interface.yaml'],human_rDNA,PRO,/Users/mstolarczyk/SRR8608074_PE1.fastq.gz,PAIRED,"HEK w/ osTIR1, ZNF143AID PRO-seq",HEK_PRO-seq,SRR8608074,Auxin,8,/Users/mstolarczyk/SRR8608074_PE2.fastq.gz
HEK_ARF_PRO-seq,hg38,human,['$CODE/peppro/sample_pipeline_interface.yaml'],human_rDNA,PRO,/Users/mstolarczyk/SRR8608070_PE1.fastq.gz,PAIRED,"HEK w/ osTIR1, ZNF143AID, ARF PRO-seq",HEK_ARF_PRO-seq,SRR8608070,Auxin,8,/Users/mstolarczyk/SRR8608070_PE2.fastq.gz
H9_PRO-seq_1,hg38,human,['$CODE/peppro/sample_pipeline_interface.yaml'],human_rDNA,PRO,/Users/mstolarczyk/H9_DMSO_rep1_PE1.fastq.gz,PAIRED,H9 PRO-seq,H9_PRO-seq_1,H9_DMSO_rep1,DMSO,8,/Users/mstolarczyk/H9_DMSO_rep1_PE2.fastq.gz
H9_PRO-seq_2,hg38,human,['$CODE/peppro/sample_pipeline_interface.yaml'],human_rDNA,PRO,/Users/mstolarczyk/H9_DMSO_rep2_PE1.fastq.gz,PAIRED,H9 PRO-seq,H9_PRO-seq_2,H9_DMSO_rep2,DMSO,8,/Users/mstolarczyk/H9_DMSO_rep2_PE2.fastq.gz
H9_PRO-seq_3,hg38,human,['$CODE/peppro/sample_pipeline_interface.yaml'],human_rDNA,PRO,/Users/mstolarczyk/H9_DMSO_rep3_PE1.fastq.gz,PAIRED,H9 PRO-seq,H9_PRO-seq_3,H9_DMSO_rep3,DMSO,8,/Users/mstolarczyk/H9_DMSO_rep3_PE2.fastq.gz
H9_treated_PRO-seq_1,hg38,human,['$CODE/peppro/sample_pipeline_interface.yaml'],human_rDNA,PRO,/Users/mstolarczyk/H9_200nM_romidepsin_rep1_PE1.fastq.gz,PAIRED,H9 treated PRO-seq,H9_treated_PRO-seq_1,H9_200nM_romidepsin_rep1,200 nM romidepsin,8,/Users/mstolarczyk/H9_200nM_romidepsin_rep1_PE2.fastq.gz
H9_treated_PRO-seq_2,hg38,human,['$CODE/peppro/sample_pipeline_interface.yaml'],human_rDNA,PRO,/Users/mstolarczyk/H9_200nM_romidepsin_rep2_PE1.fastq.gz,PAIRED,H9 treated PRO-seq,H9_treated_PRO-seq_2,H9_200nM_romidepsin_rep2,200 nM romidepsin,8,/Users/mstolarczyk/H9_200nM_romidepsin_rep2_PE2.fastq.gz
H9_treated_PRO-seq_3,hg38,human,['$CODE/peppro/sample_pipeline_interface.yaml'],human_rDNA,PRO,/Users/mstolarczyk/H9_200nM_romidepsin_rep3_PE1.fastq.gz,PAIRED,H9 treated PRO-seq,H9_treated_PRO-seq_3,H9_200nM_romidepsin_rep3,200 nM romidepsin,8,/Users/mstolarczyk/H9_200nM_romidepsin_rep3_PE2.fastq.gz




H9_PRO-seq_10,hg38,human,['$CODE/peppro/sample_pipeline_interface.yaml'],human_rDNA,PRO,/Users/mstolarczyk/H9_PRO-seq_10pct_PE1.fastq.gz,PAIRED,10% subset H9 PRO-seq 2,H9_PRO-seq_10,H9_PRO-seq_10pct,DMSO,8,/Users/mstolarczyk/H9_PRO-seq_10pct_PE2.fastq.gz
H9_PRO-seq_20,hg38,human,['$CODE/peppro/sample_pipeline_interface.yaml'],human_rDNA,PRO,/Users/mstolarczyk/H9_PRO-seq_20pct_PE1.fastq.gz,PAIRED,20% subset H9 PRO-seq 2,H9_PRO-seq_20,H9_PRO-seq_20pct,DMSO,8,/Users/mstolarczyk/H9_PRO-seq_20pct_PE2.fastq.gz
H9_PRO-seq_30,hg38,human,['$CODE/peppro/sample_pipeline_interface.yaml'],human_rDNA,PRO,/Users/mstolarczyk/H9_PRO-seq_30pct_PE1.fastq.gz,PAIRED,30% subset H9 PRO-seq 2,H9_PRO-seq_30,H9_PRO-seq_30pct,DMSO,8,/Users/mstolarczyk/H9_PRO-seq_30pct_PE2.fastq.gz
H9_PRO-seq_40,hg38,human,['$CODE/peppro/sample_pipeline_interface.yaml'],human_rDNA,PRO,/Users/mstolarczyk/H9_PRO-seq_40pct_PE1.fastq.gz,PAIRED,40% subset H9 PRO-seq 2,H9_PRO-seq_40,H9_PRO-seq_40pct,DMSO,8,/Users/mstolarczyk/H9_PRO-seq_40pct_PE2.fastq.gz
H9_PRO-seq_50,hg38,human,['$CODE/peppro/sample_pipeline_interface.yaml'],human_rDNA,PRO,/Users/mstolarczyk/H9_PRO-seq_50pct_PE1.fastq.gz,PAIRED,50% subset H9 PRO-seq 2,H9_PRO-seq_50,H9_PRO-seq_50pct,DMSO,8,/Users/mstolarczyk/H9_PRO-seq_50pct_PE2.fastq.gz
H9_PRO-seq_60,hg38,human,['$CODE/peppro/sample_pipeline_interface.yaml'],human_rDNA,PRO,/Users/mstolarczyk/H9_PRO-seq_60pct_PE1.fastq.gz,PAIRED,60% subset H9 PRO-seq 2,H9_PRO-seq_60,H9_PRO-seq_60pct,DMSO,8,/Users/mstolarczyk/H9_PRO-seq_60pct_PE2.fastq.gz
H9_PRO-seq_70,hg38,human,['$CODE/peppro/sample_pipeline_interface.yaml'],human_rDNA,PRO,/Users/mstolarczyk/H9_PRO-seq_70pct_PE1.fastq.gz,PAIRED,70% subset H9 PRO-seq 2,H9_PRO-seq_70,H9_PRO-seq_70pct,DMSO,8,/Users/mstolarczyk/H9_PRO-seq_70pct_PE2.fastq.gz
H9_PRO-seq_80,hg38,human,['$CODE/peppro/sample_pipeline_interface.yaml'],human_rDNA,PRO,/Users/mstolarczyk/H9_PRO-seq_80pct_PE1.fastq.gz,PAIRED,80% subset H9 PRO-seq 2,H9_PRO-seq_80,H9_PRO-seq_80pct,DMSO,8,/Users/mstolarczyk/H9_PRO-seq_80pct_PE2.fastq.gz
H9_PRO-seq_90,hg38,human,['$CODE/peppro/sample_pipeline_interface.yaml'],human_rDNA,PRO,/Users/mstolarczyk/H9_PRO-seq_90pct_PE1.fastq.gz,PAIRED,90% subset H9 PRO-seq 2,H9_PRO-seq_90,H9_PRO-seq_90pct,DMSO,8,/Users/mstolarczyk/H9_PRO-seq_90pct_PE2.fastq.gz
H9_PRO-seq_100,hg38,human,['$CODE/peppro/sample_pipeline_interface.yaml'],human_rDNA,PRO,/Users/mstolarczyk/H9_PRO-seq_100pct_PE1.fastq.gz,PAIRED,100% H9 PRO-seq 2,H9_PRO-seq_100,H9_PRO-seq_100pct,DMSO,8,/Users/mstolarczyk/H9_PRO-seq_100pct_PE2.fastq.gz
Terminal window