Awesome Pipeline

A curated list of awesome pipeline toolkits inspired by Awesome Sysadmin

Pipeline frameworks & libraries

• ActionChain - A workflow system for simple linear success/failure workflows.
• Adage - Small package to describe workflows that are not completely known at definition time.
• AiiDA - workflow manager with a strong focus on provenance, performance and extensibility.
• Airflow - Python-based workflow system created by AirBnb.
• Anduril - Component-based workflow framework for scientific data analysis.
• Antha - High-level language for biology.
• Argo Workflows - Container-native workflow engine for orchestrating parallel data processing, ML, or CI jobs on Kubernetes.
• Autosubmit - An open source Python experiment and workflow manager used to manage complex workflows on Cloud and HPC platforms.
• AWE - Workflow and resource management system with CWL support.
• Balsam - Python-based high throughput task and workflow engine.
• Bds - Scripting language for data pipelines.
• Beam - Unified programming model for batch and streaming data-parallel processing pipelines.
• BioMake - GNU-Make-like utility for managing builds and complex workflows.
• BioQueue - Explicit framework with web monitoring and resource estimation.
• Bioshake - Haskell DSL built on shake with strong typing and EDAM support.
• Bistro - Library to build and execute typed scientific workflows.
• Bpipe - Tool for running and managing bioinformatics pipelines.
• Briefly - Python Meta-programming Library for Job Flow Control.
• Burr - Python based lightweight graph (i.e. can do loops and conditional branching, and not just DAGs) orchestrator.
• Cluster Flow - Command-line tool which uses common cluster managers to run bioinformatics pipelines.
• Clusterjob - Automated reproducibility, and hassle-free submission of computational jobs to clusters.
• Cocoindex - ETL framework to build fresh index.
• Compi - Application framework for portable computational pipelines.
• Compss - Programming model for distributed infrastructures.
• Conan2 - Light-weight workflow management application.
• Consecution - A Python pipeline abstraction inspired by Apache Storm topologies.
• Cosmos - Python library for massively parallel workflows.
• Couler - Unified interface for constructing and managing workflows on different workflow engines, such as Argo Workflows, Tekton Pipelines, and Apache Airflow.
• Covalent - Workflow orchestration toolkit for high-performance and quantum computing research and development.
• Cromwell - Workflow Management System geared towards scientific workflows from the Broad Institute.
• Cuneiform