Pipeline Orchestration — professional stack.
For each requirement below, pick the option that fits your build — recommended first, then free and cheaper alternatives — or skip what your project doesn't need. Tap the info icon next to any requirement to see why it matters.
Orchestrator Core
A piece of your stack you may or may not need, depending on scope. Pick the option that fits — or skip it if your project doesn’t require this capability yet.Pricing & free-tier limitsAirflow OSS Free self-hosted | MWAA $0.55/hr env (~$396/mo) + $0.013/vCPU-hr + $0.0068/GB-hr | Astronomer Standard $550/mo per env + $0.18 per Astro Unit-hr, Scale $2000/mo | Dagster Cloud Free 1M events/mo + 3 seats, Pro $300/mo + $0.25 per 1K runs over, Insights $0.10 per 10K events overage
Scheduler / Cron Engine
Runs slow work (emails, exports, payouts) in the background so users don’t wait. Keeps your app snappy.Pricing & free-tier limitsLinux Cron Free | GitHub Actions Cron 2K mins/mo Free then $0.008/min | EventBridge Scheduler Free 14M invocations then $1.00 per 1M reqs over | Temporal Cloud $25/mo + $0.20 per 1K scheduled actions over 100K
Metadata Database (Airflow / Dagster metastore)
Persistent storage for your app’s data — users, products, orders. The single most important architectural decision for most projects.Pricing & free-tier limitsSelf-hosted Free + compute | Neon Free 0.5GB storage + 100 CU-hrs/mo then Scale $19/mo 10GB + $0.12/CU-hr over | Supabase Free 500MB then $25/mo 8GB + $0.21/GB over | RDS db.t3.micro $15/mo, db.r6g.large $220/mo + $0.115/GB-mo storage, Snapshot $0.095/GB-mo
Worker Execution / Compute (Celery / K8s Executor)
Where your code actually runs and serves requests. Picking the right host affects speed, scaling, and how much ops work you do.Pricing & free-tier limitsDocker Compose Free | ECS Fargate Spot $0.0124/vCPU-hr + $0.00138/GB-hr (70% off) | EKS Control Plane $73/mo + EC2 Spot m6i.large $0.03/hr, Fargate $0.04/vCPU-hr | GKE Autopilot $0.0445/vCPU-hr + $0.0049/GB-hr + $0.10/hr mgmt fee
Artifact Storage for DAG Definitions & Code
Where you keep files users upload or you serve — images, videos, documents — and how fast they reach visitors around the world.Pricing & free-tier limitsLocal FS Free | MinIO OSS Free + disk | R2 Free 10GB storage + 10M Class A + 50M Class B ops/mo, then $0.015/GB-mo storage | S3 $0.023/GB-mo first 50TB, PUT $0.005/1K, GET $0.0004/1K, Egress $0.09/GB first 10TB over free tier
Log Aggregation for Task Logs
Tells you when things break and why. You cannot fix what you cannot see — this is how you keep downtime short.Pricing & free-tier limitsLoki OSS Free self-hosted | CloudWatch Free 5GB ingest + 5GB archive then $0.50/GB ingest + $0.03/GB-mo archive, $0.50/GB scan Logs Insights | Datadog Free 1GB/day trial then $0.10/GB ingest + $1.70 per 1M events indexed over 100, + $0.10/GB retention over
Monitoring for DAG Runs (Uptime, Duration)
Tells you when things break and why. You cannot fix what you cannot see — this is how you keep downtime short.Pricing & free-tier limitsPrometheus/Grafana OSS Free | Grafana Cloud Free 10K metrics + 50GB logs + 50GB traces then Pro $8/user/mo + $8 per 1K metrics over | AMP $0.90 per 1M ingested samples free 40M samples query, AMG $9/editor/mo | Datadog Infra Enterprise $23/host/mo + APM $34/host-mo + $0.20/1K spans overage
Alerting / Incident Management
Tells you when things break and why. You cannot fix what you cannot see — this is how you keep downtime short.Pricing & free-tier limitsSlack Webhook Free | SES Free 3K msgs/mo then $0.10 per 1K | Opsgenie Free 5 users + 1K SMS mins then Standard $29/user/mo + $0.30/SMS overage, Enterprise $71/user/mo | PagerDuty Free 1 user, Pro $41/user/mo + $0.25/SMS + $0.10/email over 500/mo
Secrets Management for Pipeline Credentials
A piece of your stack you may or may not need, depending on scope. Pick the option that fits — or skip it if your project doesn’t require this capability yet.Pricing & free-tier limitsSOPS/Env Free | Doppler Free 25 secrets 1 project then Team $18/user/mo 500 secrets + $0.05/secret over | AWS Secrets $0.40/secret/mo + $0.05 per 10K API calls, Free 30 days | HCP Vault $0.03/hr cluster ~$22/mo + $0.03 per 10K txns, Dedicated $0.48/hr ~$350/mo
CI/CD for DAG Deployment
Automates testing and deploying your code. Saves enormous time and prevents “works on my machine” releases.Pricing & free-tier limitsJenkins OSS Free | GitHub Actions Free 2000 Linux mins + 500MB storage then $0.008/min Linux + $0.25/GB-mo overage | GitLab Free 400 CI mins then Premium $29/user/mo + $0.008/min over | CircleCI Free 6000 credits then Scale $35/mo + $0.06/credit over, $0.10/GB egress over
Backup for Orchestrator Metadata DB
Copies of your data so a bug, hack, or outage doesn’t become permanent loss. Cheap insurance every serious project needs.Pricing & free-tier limitspg_dump Free + S3 storage $0.023/GB-mo | AWS Backup $0.05/GB-mo warm backup + $0.02/GB-mo warm + $0.03 restore over | RDS Snapshot Free up to DB size 7-day retention + $0.095/GB-mo over, PITR storage $0.095/GB-mo | Veeam $18 per instance/mo + $0.05/GB-mo overage
Dependency Management & Lineage Tracking
Rates, labels, and tracking for physical goods. Needed once you actually ship products.Pricing & free-tier limitsdbt Core/Marquez OSS Free | dbt Cloud Developer Free 1 seat + 300 models/mo then Team $100/user/mo + $0.02 per model run over 5000 | DataHub Free 1M events then Team $250/mo + $0.30 per 1K ops over | Atlan Team $25K/yr base + $0.50/GB lineage storage over
Message Queue / Event Bus (Event-driven DAG triggers)
Runs slow work (emails, exports, payouts) in the background so users don’t wait. Keeps your app snappy.Pricing & free-tier limitsRabbitMQ/Redis OSS Free | SQS Free 1M req/mo then $0.40 per 1M + $0.50/M SNS, EventBridge $1 per 1M events over free 14M | Confluent Standard $0.15/CCU-hr (~$108/mo) + $0.20/GB ingress + $0.11/GB egress over, storage $0.10/GB-mo
Container Registry for Custom Worker Images
Runs slow work (emails, exports, payouts) in the background so users don’t wait. Keeps your app snappy.Pricing & free-tier limitsGHCR Free 500MB storage + 1GB transfer, Docker Hub Free 1 private repo + 200 pulls/6hr then Pro $5/mo | ECR $0.10/GB-mo storage 500MB free, $0.09/GB egress 1GB free, $0.20 per 10K API calls over free, transfer $0.09/GB over
Version Control Hosting for DAGs (Git)
Where your code actually runs and serves requests. Picking the right host affects speed, scaling, and how much ops work you do.Pricing & free-tier limitsGitHub Free unlimited repos 500MB packages + 2000 Actions mins then Team $4/user/mo 3000 mins + $0.008/min over | Bitbucket Free 5 users then $3.30/user/mo + $0.01/min over | GitHub Enterprise $21/user/mo + 50K mins, $0.01 storage/GB over
IaC Provisioning for Orchestrator (Terraform)
Enterprise single sign-on and user provisioning. Usually required to sell to larger organizations.Pricing & free-tier limitsTerraform OSS/OpenTofu Free | Pulumi Free 1 org + 1000 mins + 1M managed hrs then Team $75/mo 10 seats + $0.10/cred-hr over | Terraform Cloud Free 500 resources then Team $70/mo + $0.00014/resource-hr over 2500, Business $149/org/mo + $0.003/state over 1000 runs
Metrics Store (Task Duration, Success Rate, Latency)
A piece of your stack you may or may not need, depending on scope. Pick the option that fits — or skip it if your project doesn’t require this capability yet.Pricing & free-tier limitsPrometheus OSS Free | Grafana Cloud Free 10K metrics + 50GB logs then Pro $8/user/mo + $8 per 1K active metrics over 15K + $0.50/GB ingest over | AMP Free 40M samples query then $0.90/M ingested over + $0.04/M query over | Datadog Metrics $0.05 per 100 custom metrics over 100 included
State Store / XCom Backend
Where your code actually runs and serves requests. Picking the right host affects speed, scaling, and how much ops work you do.Pricing & free-tier limitsPostgres XCom Free | Redis OSS Free | ElastiCache Serverless Free 500MB then $0.30/GB-hr + $0.20 per 1M ECPUs over free 1M | DynamoDB Free 25GB + 25 RCU/WCU then $0.25/GB-mo + $0.471 per M WCU over, Redis Cloud $5/mo 100MB then Pro $0.90/GB-hr + $0.10/10K ops over
Distributed Lock / Concurrency Control
A piece of your stack you may or may not need, depending on scope. Pick the option that fits — or skip it if your project doesn’t require this capability yet.Pricing & free-tier limitsPostgres locks Free | Redis OSS/etcd OSS Free | DynamoDB Free 25GB + 25 WCU then $0.25/GB-mo + $0.47/M writes over 1M | Aiven Kafka+ZK Startup $129/mo 2GB RAM, HCP Consul $0.05/hr cluster ~$36/mo + $0.10 per 10K reads over
API Gateway for Pipeline Triggers (REST/Webhook)
Lets external services notify you of events in real time. Core to integrating payments, KYC, and third-party workflows.Pricing & free-tier limitsNGINX/Kong OSS Free | Cloudflare Workers Free 100K req/day then $0.15/M over | API Gateway HTTP Free 1M calls/mo 12mo then $1.00 per M + $0.09/GB + $3.50/M cache over | Kong Enterprise $250/mo per gateway node + $0.025 per 1K reqs over 5M included
Authentication & RBAC (SSO for Orchestrator UI)
How users sign up, log in, and are authorized. Getting roles and access control right early prevents painful rewrites.Pricing & free-tier limitsKeycloak OSS Free | Auth0 Free 7.5K MAU + unlimited machine-to-machine then $35/mo 500 MAU + $0.15 per MAU over 1K + $0.02 per MFA over | Okta Starter $200/mo 250 MAU + $8 per workforce user/mo + $0.35 per MAU over, Enterprise from $850/mo
Data Quality Validation Infrastructure
A piece of your stack you may or may not need, depending on scope. Pick the option that fits — or skip it if your project doesn’t require this capability yet.Pricing & free-tier limitsGE/Soda Core/Deequ OSS Free | Soda Cloud Free 10 datasets + 100 tests then Team $500/mo 50 datasets + $10 per dataset over | Elementary Cloud Free 2 env + Team $360/mo + $0.15 per test over 5K | Monte Carlo $35K/yr starter 250 tables + $70 per extra table, $0.20/GB scanned overage
Audit Logs WORM for Pipeline Executions (Compliance)
Keeps you secure and compliant with regulations. Required for enterprise customers and handling sensitive data.Pricing & free-tier limitsS3 Object Lock Free feature + storage $0.023/GB-mo normal, PUT $0.005/1K | CloudTrail First trail Free then $2.00 per 100K mgmt events over, Lake $0.20/GB ingested + $0.023/GB-mo storage 7yr retention + $0.005 per GB query scanned over 10GB
Service Discovery / Internal DNS for Workers
Your human-readable address on the web plus the DNS records that point it at your servers. Skipping this is not an option — users need a domain to reach you.Pricing & free-tier limitsCoreDNS/Kube-DNS Free | Cloud Map $0.10 per 1K Discover API calls over free 1M then $0.10 per namespace-mo + Route53 PHZ $0.50/zone/mo + $0.40 per M queries over | Consul HCP $0.03/hr cluster ~$22/mo + $0.10 per 10K reads over free 1M
Auto-scaling for Workers (KEDA / HPA / Karpenter)
A piece of your stack you may or may not need, depending on scope. Pick the option that fits — or skip it if your project doesn’t require this capability yet.Pricing & free-tier limitsKEDA/HPA OSS Free | ECS Auto Scaling Free (pay compute Fargate $0.04/vCPU-hr) | Spot.io Ocean Free 1 cluster up to 20 vCPU then $0.018/vCPU-hr fee on savings + compute, overage $0.02/vCPU-hr, Karpenter OSS Free + EC2 spot savings up to 70%
Tracing (OpenTelemetry for DAG Tasks)
Tells you when things break and why. You cannot fix what you cannot see — this is how you keep downtime short.Pricing & free-tier limitsJaeger/Tempo/Zipkin OSS Free | X-Ray Free 100K traces/mo + 500K service map invocations then $5.00 per 1M traces over, $0.01 per 100K retrievals | Honeycomb Free 20M events/mo then Pro $100/mo includes 100M events + $0.10 per M over | Datadog APM $0.20 per 1K spans over 1K included per host
Dead Letter Queue & Retry Infrastructure
Runs slow work (emails, exports, payouts) in the background so users don’t wait. Keeps your app snappy.Pricing & free-tier limitsRabbitMQ DLX Free | SQS DLQ Free tier same as SQS 1M req/mo free then $0.40 per M reqs over | Step Functions $0.025 per 1K state transitions free 4K/mo then $0.025 per 1K over | Temporal Cloud DLQ included, Actions $0.00002 each over 5M, Retries free
Environment Management (dev/staging/prod orchestrators)
Stores API keys and credentials safely, separate from your code. Leaking secrets is one of the most common breaches.Pricing & free-tier limitsCompose/Env Free | GitHub Environments Free on public + Team $4/user/mo + protected envs | Doppler Free 25 secrets 1 env then Team $18/user/mo unlimited envs + $0.06/env over 20 | Harness $50/user/mo Pro + $200/mo per env pack over 10 envs, Spacelift $200/mo + $0.05 per runner min over
Cost Tracking & Attribution (FinOps for Pipelines)
Rates, labels, and tracking for physical goods. Needed once you actually ship products.Pricing & free-tier limitsOpenCost Free | Kubecost Free 1 cluster 15-day retention then Team $600/mo per cluster up to 20K container-hrs + $0.05/container-hr over | AWS CE Free + CU Reports $0.01/1K lines + Data Exports $0.01/GB over | CloudHealth Trial then $0.01 per $1 AWS spend min $2K/mo + $0.50 per resource over 10K
Data Catalog / Metadata Platform Hook
A piece of your stack you may or may not need, depending on scope. Pick the option that fits — or skip it if your project doesn’t require this capability yet.Pricing & free-tier limitsOpenMetadata/DataHub OSS Free | DataHub Cloud Free 1M metadata ops + 3 users then Team $250/mo 5 users + $0.30 per 1K ops over 5M + $20 per extra user | Atlan Team $25K/yr base + $0.50 per query log GB over 100GB + $1500 per extra connector over 10
How this professional pipeline orchestration checklist works.
Each requirement below is something a professional pipeline orchestration build typically needs. Pick one of the four researched options — recommended, free, cheaper or paid — add your own with "Other", or skip the requirement if your project doesn't need it. Nothing is mandatory; the plan on the right tracks what you've decided so nothing gets forgotten.
Your picks are saved in this browser automatically, so you can come back anytime. Options are researched per build level and refreshed as vendors change their plans — always verify details on the provider's page before committing.