Pipeline Orchestration — small scale stack.
For each requirement below, pick the option that fits your build — recommended first, then free and cheaper alternatives — or skip what your project doesn't need. Tap the info icon next to any requirement to see why it matters.
Orchestrator Core
A piece of your stack you may or may not need, depending on scope. Pick the option that fits — or skip it if your project doesn’t require this capability yet.Pricing & free-tier limitsAirflow OSS Free self-hosted | MWAA $0.55/hr env (~$396/mo) + $0.013/vCPU-hr + $0.0068/GB-hr | Astronomer Standard $550/mo per env + $0.18 per Astro Unit-hr, Scale $2000/mo | Dagster Cloud Free 1M events/mo + 3 seats, Pro $300/mo + $0.25 per 1K runs over, Insights $0.10 per 10K events overage
Scheduler / Cron Engine
Runs slow work (emails, exports, payouts) in the background so users don’t wait. Keeps your app snappy.Pricing & free-tier limitsLinux Cron Free | GitHub Actions Cron 2K mins/mo Free then $0.008/min | EventBridge Scheduler Free 14M invocations then $1.00 per 1M reqs over | Temporal Cloud $25/mo + $0.20 per 1K scheduled actions over 100K
Metadata Database (Airflow / Dagster metastore)
Persistent storage for your app’s data — users, products, orders. The single most important architectural decision for most projects.Pricing & free-tier limitsSelf-hosted Free + compute | Neon Free 0.5GB storage + 100 CU-hrs/mo then Scale $19/mo 10GB + $0.12/CU-hr over | Supabase Free 500MB then $25/mo 8GB + $0.21/GB over | RDS db.t3.micro $15/mo, db.r6g.large $220/mo + $0.115/GB-mo storage, Snapshot $0.095/GB-mo
Worker Execution / Compute (Celery / K8s Executor)
Where your code actually runs and serves requests. Picking the right host affects speed, scaling, and how much ops work you do.Pricing & free-tier limitsDocker Compose Free | ECS Fargate Spot $0.0124/vCPU-hr + $0.00138/GB-hr (70% off) | EKS Control Plane $73/mo + EC2 Spot m6i.large $0.03/hr, Fargate $0.04/vCPU-hr | GKE Autopilot $0.0445/vCPU-hr + $0.0049/GB-hr + $0.10/hr mgmt fee
Artifact Storage for DAG Definitions & Code
Where you keep files users upload or you serve — images, videos, documents — and how fast they reach visitors around the world.Pricing & free-tier limitsLocal FS Free | MinIO OSS Free + disk | R2 Free 10GB storage + 10M Class A + 50M Class B ops/mo, then $0.015/GB-mo storage | S3 $0.023/GB-mo first 50TB, PUT $0.005/1K, GET $0.0004/1K, Egress $0.09/GB first 10TB over free tier
Log Aggregation for Task Logs
Tells you when things break and why. You cannot fix what you cannot see — this is how you keep downtime short.Pricing & free-tier limitsLoki OSS Free self-hosted | CloudWatch Free 5GB ingest + 5GB archive then $0.50/GB ingest + $0.03/GB-mo archive, $0.50/GB scan Logs Insights | Datadog Free 1GB/day trial then $0.10/GB ingest + $1.70 per 1M events indexed over 100, + $0.10/GB retention over
Monitoring for DAG Runs (Uptime, Duration)
Tells you when things break and why. You cannot fix what you cannot see — this is how you keep downtime short.Pricing & free-tier limitsPrometheus/Grafana OSS Free | Grafana Cloud Free 10K metrics + 50GB logs + 50GB traces then Pro $8/user/mo + $8 per 1K metrics over | AMP $0.90 per 1M ingested samples free 40M samples query, AMG $9/editor/mo | Datadog Infra Enterprise $23/host/mo + APM $34/host-mo + $0.20/1K spans overage
Alerting / Incident Management
Tells you when things break and why. You cannot fix what you cannot see — this is how you keep downtime short.Pricing & free-tier limitsSlack Webhook Free | SES Free 3K msgs/mo then $0.10 per 1K | Opsgenie Free 5 users + 1K SMS mins then Standard $29/user/mo + $0.30/SMS overage, Enterprise $71/user/mo | PagerDuty Free 1 user, Pro $41/user/mo + $0.25/SMS + $0.10/email over 500/mo
Secrets Management for Pipeline Credentials
A piece of your stack you may or may not need, depending on scope. Pick the option that fits — or skip it if your project doesn’t require this capability yet.Pricing & free-tier limitsSOPS/Env Free | Doppler Free 25 secrets 1 project then Team $18/user/mo 500 secrets + $0.05/secret over | AWS Secrets $0.40/secret/mo + $0.05 per 10K API calls, Free 30 days | HCP Vault $0.03/hr cluster ~$22/mo + $0.03 per 10K txns, Dedicated $0.48/hr ~$350/mo
CI/CD for DAG Deployment
Automates testing and deploying your code. Saves enormous time and prevents “works on my machine” releases.Pricing & free-tier limitsJenkins OSS Free | GitHub Actions Free 2000 Linux mins + 500MB storage then $0.008/min Linux + $0.25/GB-mo overage | GitLab Free 400 CI mins then Premium $29/user/mo + $0.008/min over | CircleCI Free 6000 credits then Scale $35/mo + $0.06/credit over, $0.10/GB egress over
Backup for Orchestrator Metadata DB
Copies of your data so a bug, hack, or outage doesn’t become permanent loss. Cheap insurance every serious project needs.Pricing & free-tier limitspg_dump Free + S3 storage $0.023/GB-mo | AWS Backup $0.05/GB-mo warm backup + $0.02/GB-mo warm + $0.03 restore over | RDS Snapshot Free up to DB size 7-day retention + $0.095/GB-mo over, PITR storage $0.095/GB-mo | Veeam $18 per instance/mo + $0.05/GB-mo overage
Dependency Management & Lineage Tracking
Rates, labels, and tracking for physical goods. Needed once you actually ship products.Pricing & free-tier limitsdbt Core/Marquez OSS Free | dbt Cloud Developer Free 1 seat + 300 models/mo then Team $100/user/mo + $0.02 per model run over 5000 | DataHub Free 1M events then Team $250/mo + $0.30 per 1K ops over | Atlan Team $25K/yr base + $0.50/GB lineage storage over
Message Queue / Event Bus (Event-driven DAG triggers)
Runs slow work (emails, exports, payouts) in the background so users don’t wait. Keeps your app snappy.Pricing & free-tier limitsRabbitMQ/Redis OSS Free | SQS Free 1M req/mo then $0.40 per 1M + $0.50/M SNS, EventBridge $1 per 1M events over free 14M | Confluent Standard $0.15/CCU-hr (~$108/mo) + $0.20/GB ingress + $0.11/GB egress over, storage $0.10/GB-mo
Container Registry for Custom Worker Images
Runs slow work (emails, exports, payouts) in the background so users don’t wait. Keeps your app snappy.Pricing & free-tier limitsGHCR Free 500MB storage + 1GB transfer, Docker Hub Free 1 private repo + 200 pulls/6hr then Pro $5/mo | ECR $0.10/GB-mo storage 500MB free, $0.09/GB egress 1GB free, $0.20 per 10K API calls over free, transfer $0.09/GB over
Version Control Hosting for DAGs (Git)
Where your code actually runs and serves requests. Picking the right host affects speed, scaling, and how much ops work you do.Pricing & free-tier limitsGitHub Free unlimited repos 500MB packages + 2000 Actions mins then Team $4/user/mo 3000 mins + $0.008/min over | Bitbucket Free 5 users then $3.30/user/mo + $0.01/min over | GitHub Enterprise $21/user/mo + 50K mins, $0.01 storage/GB over
IaC Provisioning for Orchestrator (Terraform)
Enterprise single sign-on and user provisioning. Usually required to sell to larger organizations.Pricing & free-tier limitsTerraform OSS/OpenTofu Free | Pulumi Free 1 org + 1000 mins + 1M managed hrs then Team $75/mo 10 seats + $0.10/cred-hr over | Terraform Cloud Free 500 resources then Team $70/mo + $0.00014/resource-hr over 2500, Business $149/org/mo + $0.003/state over 1000 runs
Metrics Store (Task Duration, Success Rate, Latency)
A piece of your stack you may or may not need, depending on scope. Pick the option that fits — or skip it if your project doesn’t require this capability yet.Pricing & free-tier limitsPrometheus OSS Free | Grafana Cloud Free 10K metrics + 50GB logs then Pro $8/user/mo + $8 per 1K active metrics over 15K + $0.50/GB ingest over | AMP Free 40M samples query then $0.90/M ingested over + $0.04/M query over | Datadog Metrics $0.05 per 100 custom metrics over 100 included
State Store / XCom Backend
Where your code actually runs and serves requests. Picking the right host affects speed, scaling, and how much ops work you do.Pricing & free-tier limitsPostgres XCom Free | Redis OSS Free | ElastiCache Serverless Free 500MB then $0.30/GB-hr + $0.20 per 1M ECPUs over free 1M | DynamoDB Free 25GB + 25 RCU/WCU then $0.25/GB-mo + $0.471 per M WCU over, Redis Cloud $5/mo 100MB then Pro $0.90/GB-hr + $0.10/10K ops over
Distributed Lock / Concurrency Control
A piece of your stack you may or may not need, depending on scope. Pick the option that fits — or skip it if your project doesn’t require this capability yet.Pricing & free-tier limitsPostgres locks Free | Redis OSS/etcd OSS Free | DynamoDB Free 25GB + 25 WCU then $0.25/GB-mo + $0.47/M writes over 1M | Aiven Kafka+ZK Startup $129/mo 2GB RAM, HCP Consul $0.05/hr cluster ~$36/mo + $0.10 per 10K reads over
How this small scale pipeline orchestration checklist works.
Each requirement below is something a small scale pipeline orchestration build typically needs. Pick one of the four researched options — recommended, free, cheaper or paid — add your own with "Other", or skip the requirement if your project doesn't need it. Nothing is mandatory; the plan on the right tracks what you've decided so nothing gets forgotten.
Your picks are saved in this browser automatically, so you can come back anytime. Options are researched per build level and refreshed as vendors change their plans — always verify details on the provider's page before committing.