Archiving namespaces
Creating an archive of specified namespaces
Section titled “Creating an archive of specified namespaces”Developers can create an archive of specified namespaces by following these steps:
1. Set environment variables for the database connection.
Section titled “1. Set environment variables for the database connection.”Env file can be found here: https://github.com/pepkit/geopephub/blob/main/environment/dev.env
2. Install geopephub package:
Section titled “2. Install geopephub package:”pip install geopephub3. Use geopephub help:
Section titled “3. Use geopephub help:”geopephub auto-download --helphelp
╭─ Commands ───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮│ auto-download Automatically download projects from geo namespace, tar them and upload to s3. Don't forget to set up AWS credentials ││ bedbase-stats Download projects from the bedbase namespace and save them to the database ││ check-by-date Check if all projects were uploaded successfully in specified period and upload them if not. Additionally, you can download projects from huge period of time. e.g. if you want to download projects from ││ check-by-date Check if all projects were uploaded successfully in specified period and upload them if not. Additionally, you can download projects from huge period of time. e.g. if you want to download projects from ││ 2020/02/25 to 2021/05/27, you should set start_period=2020/02/25, and end_period=2021/05/27 ││ clean-history Clean history of the pephub ││ download Download projects from the particular namespace. You can filter projects, order them, and download only part of them. ││ run-checker Check if all projects were uploaded successfully in specified period and upload them if not. To check if all projects were uploaded successfully 3 periods ago, where one period is 1 day, and cycles are ││ happening every day, you should set cycle_count=3, and period_length=1. (geopephub run_checker --cycle-count 3 --period-length 1) ││ run-queuer Queue GEO projects that were uploaded or updated in the last period ││ run-uploader Upload projects that were queued, but not uploaded yet. │╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯To automatically download projects from the namespace, and register them in the database, use the following command
we will use auto-download command. It will download all PEPs to the chosen directory, and then tar them if needed,
and upload to the s3 bucket.
geopephub auto-download --helphelp
╭─ Options ────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮│ --namespace TEXT Namespace of the projects that have to be downloaded. Default: geo [default: geo] ││ * --destination TEXT Output directory or s3 bucket. By default set current directory [default: None] [required] ││ --compress --no-compress zip downloaded projects, default: True [default: compress] ││ --tar-all --no-tar-all tar all the downloaded projects into a single file [default: tar-all] ││ --upload-s3 --no-upload-s3 upload tar file to s3 bucket [default: upload-s3] ││ --bucket TEXT S3 bucket name [default: pephub] ││ --help Show this message and exit. │╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯--tar-all and upload-s3 is defaulted to True, so if you want to download and tar all projects, and upload to the s3 bucket,
you can skip these options.