Skip to content
- Updated for logmuse 0.3.0 compatibility
- Migrated from setup.py to pyproject.toml with hatchling backend
- Replaced Black/isort with Ruff for linting and formatting
- Updated GitHub Actions to v4/v5 with Python 3.10-3.14 CI matrix
- Removed Python 2 compatibility code and legacy packaging files
- Version now sourced from importlib.metadata (removed _version.py)
- Converted all docstrings to Google-style formatting
- Consolidated documentation into pepspec; removed local docs/ and mkdocs.yml
- Updated README badges and documentation links
clean_add() now accepts None values, enabling safe use with conditionally-assigned variables- Pipestat version displayed in pipeline version log output
- Removed magic schema discovery (
default_pipestat_output_schema())
- Test quality: added missing assertions, removed permanently-skipped tests, added shared fixtures
- Remove veracitools dependency #233
- Fixed warnings for Python >3.12
- Updated version of Python to 3.13 in pytests
- bump requirements to require pipestat>=0.11.0
- remove pipestat_project_name from PipelineManager parameters
- refactor pipestat_sample_name to pipestat_record_identifier in PipelineManager parameters
- update requirements for pipestat 0.9.0, ubiquerg 0.8.0, and yacman 0.9.3
- set
force_overwrite to default to true, Issue #209
- refactor for pipestat v0.6.0 release
- drop python 2.7
- updated requirements
- changed message_raw to be a value_dict when reporting to conform to pipestat
- added
force_overwrite to report_result and report_object
- added pipestat_pipeline_type, defaulting to sample-level
- fixed self.new_start overriding checkpoints.
- added _safe_write_to_file back into pypiper for Pepatac backwards compatibility
- A few bugs with compatibility with Python version 3.9
- Removed use2to3 for compatibility with setuptools 58
- Increased requirement for logmuse
- Sort argument outputs in logs
- Fail messages can now be a string (previously required an Exception).
- Use profile to determine total elapsed time
logging functions directly on PipelineManager- Re-export
add_logging_options from logmuse, for direct use by a pipeline author.
logger_via_cli that defaults to the strict=False behavior of the same-named function from logmuse- Use logging for pypiper-generated output.
- Fix childless processes memory monitoring issue
- Fix problems with runtime reading from pipeline profile TSV formatted according to two styles
- Fix problems running containerized executables that would sometimes hang
- Fix inaccurate elapsed time accumulation
- Fixed a bug that caused hanging when running in singularity containerized executables
- Fixed bugs with merging bamfiles using samtools
- The hashes in the pipeline profile are produced from the entire original command, even if it is a pipe
- Changed output to simplify and improve log readability
- Fixed a bug that caused an OSError removing lock files for some filesystems.
- Elevate
attmap dependency bound to require inclusion of improved path expansion behavior.
- Elevate
attmap dependency bound to require inclusion of a bugfix there.
- Improve python3 handling of integers and strings
- Fixed a bug with cleanup scripts in
dirty mode
- Restructured profile output with hash and processID, and made lock paths relative
- Streamlined some logging outputs
- Allows nested parentheses and braces for piped commands
- Fixed a bug that would have split a pipe within a braced command
- Some performance improvements for ngstk functions
- Allow
ngstk.input_to_fastq to yield gzipped fastq files
- Fixed a bug that raised exception with empty commands
- Fixed the pipeline profiling issues
- Major updates to internal systems: Switch to
attmap
- Revamped way of handling child subprocesses which should lead to more
efficient memory monitoring of piped subprocesses, and more consistent
handling of rogues subprocesses during pipeline failure.
- Added force mode to ngstk
gzip and pigz use.
- Changed documentation from sphinx to mkdocs.
- Fixed a bug with python3 output buffering
- Implement multi-target commands
- Fixed a bug that had prevented new start mode from working in certain cases.
- Allow user to change units of memory passed in with default pypiper cli.
- Point release to PyPI for README rendering.
- Simple point release update to fix PyPI landing page.
- Never echo protected-looking attribute request.
- Fixed a bug in NGSTk that caused errors for read counting functions on
MACOS. MACOS
wc returns leading whitespace, which caused these functions
to fail.
- Use
psutil to track aggregate memory usage for processes that spawn
children. This results in accurate memory records for these processes.
- Individual commands in a string of commands connected by shell pipes are
now treated as individual commands, and and monitored individually for
time and memory, and if a single component, fails, the entire string will
fail. Previously, only the final return command was recorded, as in
bash.
- Various other small improvements (like waiting checking for dynamic recover
flags)
- Fixed a bug that caused a problem for some pipelines adding groups of pypiper args.
- Improved the
run waiting method to immediately stop upon job
completion, rather than minute-increment polling. This should improve
performance particularly in pipelines with many, medium-runtime steps, and
improve accuracy of timing profiles.
- Implemented 'new start' mode.
- Improved error messages and exception handling for missing child software.
- Clarified the built-in required vs. optional args by allowing pipeline authors to specify which of the pypiper args are required. The command-line help UI now displays these correctly as 'required arguments' instead of incorrectly as 'optional arguments'.
- Corrected the sort order of added arguments, so they are listed in the help menu more naturally.
- Fixed a bug that caused an erroneous error message indicating missing pypiper args.
- Clarified the license is BSD2
- Fixed a bug that neglected to list pyyaml as a dependency
- Implemented the 'report object' function.
- Cleanup files are now relative, so a moved folder could still be cleaned.
- Fixed a bug that prevented install if pypandoc was not installed
- Fixed a bug that caused an error in containers where /proc wasn't accessible
- Package cleanup for Pypi.
- Standardize
NGSTk function naming.
- Introduce
Stage as a model for a logically related set of pipeline processing steps.
- Introduce
Pipeline framework for automated processing phase execution and checkpointing.
- Add ability to start and/or stop a pipeline at arbitrary checkpoints.
- Introduce new state for a paused/halted pipeline.
- Improve spawned process shutdown to avoid zombie processes.
- Adds 'dynamic recovery' capability. For jobs that are terminated by an interrupt, such as a SIGINT or SIGTERM (as opposed to a failed command), pypiper will now set a dynamic recovery flags. These jobs, when restarted, will automatically pick up where they left off, without requiring any user intervention. Previously, the user would have to specify recover mode (
-R). Now, recover mode forces a recover regardless of failure type, but interrupted pipelines will auto-recover.
- Pypiper now appropriately adds cleanup files intermediate files for failed runs. It adds them to the cleanup script.
- Improves error messages so only a single exception is raised with a more direct relevance to the user/
- Pypiper will automatically remove existing flags when the run starts, eliminating the earlier issue of confusion due to multiple flags present on runs that were restarted.
- Fixes a bug that caused a pipeline to continue if a SIGTERM is given during a process that was marked
nofail.
- Pypiper now can handle multiple SIGTERMs without one canceling the shutdown procedure begun by the other.
- Major improvements to documentation and tutorials.
- Adds
report_figure function.
- Adds preliminary support for handling docker containers
- Updates docs, adds Hello World example
- Adds 'waiting' flag
- Eliminates extra spaces in reported results
- Pypiper module is version aware
- Updates Success time format to eliminate space
- Improves efficiency in some ngstk merging functions
- First major public release!
- Revamps pypiper args
- Adds parallel compression/decompression with pigz
- Various small bug fixes and speed improvements
