Package geofetch Documentation

Package Overview

The geofetch package provides tools for downloading metadata and data from Gene Expression Omnibus (GEO) and Sequence Read Archive (SRA). It can convert GEO/SRA metadata into PEP format for easy integration with other PEPkit tools.

Key Features

GEO/SRA Download: Fetch metadata and raw data from NCBI repositories
PEP Generation: Automatically create PEP-formatted project configs
Flexible Filtering: Search and filter GEO datasets by date and criteria
SRA Integration: Download and convert SRA data to FASTQ format
Processed Data: Download processed data matrices from GEO

Installation

pip install geofetch

Quick Example

from geofetch import Geofetcher

# Initialize geofetcher
gf = Geofetcher()

# Fetch a GEO series
gf.fetch_all(input="GSE####", name="my_project")

API Reference

Geofetcher Class

The main class for fetching data from GEO/SRA:

`Geofetcher`

class Geofetcher(name: str = '', metadata_root: str = '', metadata_folder: str = '', just_metadata: bool = False, refresh_metadata: bool = False, config_template: str | None = None, pipeline_samples: str | None = None, pipeline_project: str | None = None, skip: int = 0, acc_anno: bool = False, use_key_subset: bool = False, processed: bool = False, data_source: str = 'samples', filter: str | None = None, filter_size: str | None = None, geo_folder: str = '.', split_experiments: bool = False, bam_folder: str = '', fq_folder: str = '', sra_folder: str = '', bam_conversion: bool = False, picard_path: str = '', input: str | None = None, const_limit_project: int = 50, const_limit_discard: int = 1000, attr_limit_truncate: int = 500, max_soft_size: str = '1GB', discard_soft: bool = False, add_dotfile: bool = False, disable_progressbar: bool = False, add_convert_modifier: bool = False, opts: object | None = None, max_prefetch_size: str | int | None = None, **kwargs: object)

Class to download or get projects, metadata, data from GEO and SRA.

Constructor.

Parameters:

input (str | None): GSE number or path to the input file. (default: None)
name (str): Specify a project name. Defaults to GSE number or name of accessions file name. (default: '')
metadata_root (str): Specify a parent folder location to store metadata. The project name will be added as a subfolder (Default: $SRAMETA:). (default: '')
metadata_folder (str): Specify an absolute folder location to store metadata. No subfolder will be added. Overrides value of --metadata-root (Default: Not used (--metadata-root is used by default)). (default: '')
just_metadata (bool): If set, don't actually run downloads, just create metadata. (default: False)
refresh_metadata (bool): If set, re-download metadata even if it exists. (default: False)
config_template (str | None): Project config yaml file template. (default: None)
pipeline_samples (str | None): Specify one or more filepaths to SAMPLES pipeline interface yaml files. These will be added to the project config file to make it immediately compatible with looper. (Default: null). (default: None)
pipeline_project (str | None): Specify one or more filepaths to PROJECT pipeline interface yaml files. These will be added to the project config file to make it immediately compatible with looper. (Default: null). (default: None)
acc_anno (bool): Produce annotation sheets for each accession. Project combined PEP for the whole project won't be produced. (default: False)
discard_soft (bool): Create project without downloading soft files on the disc. (default: False)
add_dotfile (bool): Add .pep.yaml file that points .yaml PEP file. (default: False)
disable_progressbar (bool): Set true to disable progressbar. (default: False)
const_limit_project (int): Optional: Limit of the number of the constant sample characters that should not be in project yaml. (Default: 50). (default: 50)
const_limit_discard (int): Optional: Limit of the number of the constant sample characters that should not be discarded (Default: 250). (default: 1000)
attr_limit_truncate (int): Optional: Limit of the number of sample characters. Any attribute with more than X characters will truncate to the first X, where X is a number of characters (Default: 500). (default: 500)
max_soft_size (str): Optional: Max size of soft file. Supported input formats: 12B, 12KB, 12MB, 12GB. [Default value: 1GB]. (default: '1GB')
processed (bool): Download processed data (Default: download raw data). (default: False)
data_source (str): Specifies the source of data on the GEO record to retrieve processed data, which may be attached to the collective series entity, or to individual samples. Allowable values are: samples, series or both (all). Ignored unless 'processed' flag is set. (Default: samples). (default: 'samples')
filter (str | None): Filter regex for processed filenames (Default: None). Ignored unless 'processed' flag is set. (default: None)
filter_size (str | None): Filter size for processed files that are stored as sample repository (Default: None). Works only for sample data. Supported input formats: 12B, 12KB, 12MB, 12GB. Ignored unless 'processed' flag is set. (default: None)
geo_folder (str): Specify a location to store processed GEO files. Ignored unless 'processed' flag is set. (Default: $GEODATA:). (default: '.')
split_experiments (bool): Split SRR runs into individual samples. By default, SRX experiments with multiple SRR Runs will have a single entry in the annotation table, with each run as a separate row in the subannotation table. This setting instead treats each run as a separate sample [Works with raw data]. (default: False)
bam_folder (str): Optional: Specify folder of bam files. Geofetch will not download sra files when corresponding bam files already exist. (Default: $SRABAM:) [Works with raw data]. (default: '')
fq_folder (str): Optional: Specify folder of fastq files. Geofetch will not download sra files when corresponding fastq files already exist. (Default: $SRAFQ:) [Works with raw data]. (default: '')
use_key_subset (bool): Use just the keys defined in this module when writing out metadata. [Works with raw data]. (default: False)
sra_folder (str): Optional: Specify a location to store sra files. (default: '')
bam_conversion (bool): Optional: set True to convert bam files [Works with raw data]. (default: False)
picard_path (str): Specify a path to the picard jar, if you want to convert fastq to bam [Works with raw data]. (default: '')
add_convert_modifier (bool): Add looper SRA convert modifier to config file. (default: False)
skip (int): Skip some accessions. (Default: no skip). (default: 0)
opts (object | None): opts object [Optional]. (default: None)
max_prefetch_size (str | int | None): Argument to prefetch command's --max-size option. (default: None)
kwargs (object): Other values. (default: {})

Attributes

acc_anno
add_convert_modifier
add_dotfile
attr_limit_truncate
bam_conversion
bam_folder
config_template
const_limit_discard
const_limit_project
disable_progressbar
discard_soft
file_pipeline_project
file_pipeline_samples
filter_re
filter_size
fq_folder
geo_folder
just_metadata
just_object
max_prefetch_size
max_soft_size
metadata_expanded
metadata_root_full
picard_path
processed
project_name
refresh_metadata
skip
split_experiments
sra_folder
supp_by
use_key_subset

Methods

`fetch_all`

def fetch_all(input: str, name: str | None = None) -> None | peppy.Project

Main function driver/workflow.

Searches, filters, downloads and saves data and metadata from GEO and SRA.

Parameters:

input (str): GSE or input file with GSE accessions.
name (str | None): Name of the project. (default: None)

Returns:

None | peppy.Project: None or peppy Project.

`fetch_processed_one`

def fetch_processed_one(gse_file_content: list[str], gsm_file_content: list[str], gsm_filter_list: dict) -> tuple[list, list]

Fetch one processed GSE project and return its metadata.

Parameters:

gse_file_content (list[str]): GSE soft file content.
gsm_file_content (list[str]): GSM soft file content.
gsm_filter_list (dict): List of GSM that have to be downloaded.

Returns:

tuple[list, list]: Tuple of (meta_processed_samples, meta_processed_series).

`get_projects`

def get_projects(input: str, just_metadata: bool = True, discard_soft: bool = True) -> dict

Fetch projects from GEO|SRA and return peppy projects.

Parameters:

input (str): GSE number, or path to file of GSE numbers.
just_metadata (bool): Process only metadata. (default: True)
discard_soft (bool): Clean run, without downloading soft files. (default: True)

Returns:

dict: Peppy project or list of projects, if acc_anno is set.

Finder Class

Class for searching and finding GSE accessions:

`Finder`

class Finder(filters: str | None = None, retmax: int = RETMAX)

Class for finding GSE accessions in special period of time.

Initialize Finder with optional filters and result limit.

Parameters:

filters (str | None): Filters that have to be added to the query. Filter Patterns can be found here: https://www.ncbi.nlm.nih.gov/books/NBK3837/#EntrezHelp.Using_the_Advanced_Search_Pag (default: None)
retmax (int): Maximum number of retrieved accessions. (default: RETMAX)

Attributes

last_result
query_customized_ending
query_filter_str

Methods

`find_differences`

@staticmethod
def find_differences(old_list: list[str], new_list: list[str]) -> list[str]

Compare 2 lists and search for elements that are not in old list.

Parameters:

old_list (list[str]): Old list of elements.
new_list (list[str]): New list of elements.

Returns:

list[str]: List of elements that are not in old list but are in new_list.

`generate_file`

def generate_file(file_path: str, gse_list: list[str] | None = None) -> None

Save the list of GSE accessions stored in this Finder object to a given file.

Parameters:

file_path (str): Root to the file where gse accessions have to be saved.
gse_list (list[str] | None): List of gse accessions. (default: None)

`get_gse_all`

def get_gse_all() -> list[str]

Get list of all gse accessions available in GEO.

Returns:

list[str]: List of gse accessions.

`get_gse_by_date`

def get_gse_by_date(start_date: str, end_date: str | None = None) -> list[str]

Search gse accessions by providing start date and end date. By default, the last date is today.

Parameters:

start_date (str): The oldest date of update (from YYYY/MM/DD to now) [input format: 'YYYY/MM/DD'].
end_date (str | None): The nearest date of update (from __ to YYYY/MM/DD) [input format: 'YYYY/MM/DD']. (default: None)

Returns:

list[str]: List of gse accessions.

`get_gse_by_day_count`

def get_gse_by_day_count(n_days: int = 1) -> list[str]

Get list of gse accessions that were uploaded or updated in last X days.

Parameters:

n_days (int): Number of days from now [e.g. 5]. (default: 1)

Returns:

list[str]: List of gse accessions.

`get_gse_id_by_query`

def get_gse_id_by_query(url: str) -> list[str]

Run esearch (ncbi search tool) by specifying URL and retrieve gse list result.

Parameters:

url (str): URL of the query.

Returns:

list[str]: List of gse ids.

`get_gse_last_3_month`

def get_gse_last_3_month() -> list[str]

Get list of gse accessions that were uploaded or updated in last 3 months.

Returns:

list[str]: List of gse accessions.

`get_gse_last_week`

def get_gse_last_week() -> list[str]

Get list of gse accessions that were uploaded or updated in last week.

Returns:

list[str]: List of gse accessions.

`uid_to_gse`

@staticmethod
def uid_to_gse(uid: str) -> str

UID to GSE accession converter.

Parameters:

uid (str): UID string (Unique Identifier Number in GEO).

Returns:

str: GSE id string.