Pipeline Orchestration — hobby stack.
For each requirement below, pick the option that fits your build — recommended first, then free and cheaper alternatives — or skip what your project doesn't need. Tap the info icon next to any requirement to see why it matters.
Orchestrator Core
A piece of your stack you may or may not need, depending on scope. Pick the option that fits — or skip it if your project doesn’t require this capability yet.Pricing & free-tier limitsAirflow OSS Free self-hosted | MWAA $0.55/hr env (~$396/mo) + $0.013/vCPU-hr + $0.0068/GB-hr | Astronomer Standard $550/mo per env + $0.18 per Astro Unit-hr, Scale $2000/mo | Dagster Cloud Free 1M events/mo + 3 seats, Pro $300/mo + $0.25 per 1K runs over, Insights $0.10 per 10K events overage
Scheduler / Cron Engine
Runs slow work (emails, exports, payouts) in the background so users don’t wait. Keeps your app snappy.Pricing & free-tier limitsLinux Cron Free | GitHub Actions Cron 2K mins/mo Free then $0.008/min | EventBridge Scheduler Free 14M invocations then $1.00 per 1M reqs over | Temporal Cloud $25/mo + $0.20 per 1K scheduled actions over 100K
Metadata Database (Airflow / Dagster metastore)
Persistent storage for your app’s data — users, products, orders. The single most important architectural decision for most projects.Pricing & free-tier limitsSelf-hosted Free + compute | Neon Free 0.5GB storage + 100 CU-hrs/mo then Scale $19/mo 10GB + $0.12/CU-hr over | Supabase Free 500MB then $25/mo 8GB + $0.21/GB over | RDS db.t3.micro $15/mo, db.r6g.large $220/mo + $0.115/GB-mo storage, Snapshot $0.095/GB-mo
Worker Execution / Compute (Celery / K8s Executor)
Where your code actually runs and serves requests. Picking the right host affects speed, scaling, and how much ops work you do.Pricing & free-tier limitsDocker Compose Free | ECS Fargate Spot $0.0124/vCPU-hr + $0.00138/GB-hr (70% off) | EKS Control Plane $73/mo + EC2 Spot m6i.large $0.03/hr, Fargate $0.04/vCPU-hr | GKE Autopilot $0.0445/vCPU-hr + $0.0049/GB-hr + $0.10/hr mgmt fee
Artifact Storage for DAG Definitions & Code
Where you keep files users upload or you serve — images, videos, documents — and how fast they reach visitors around the world.Pricing & free-tier limitsLocal FS Free | MinIO OSS Free + disk | R2 Free 10GB storage + 10M Class A + 50M Class B ops/mo, then $0.015/GB-mo storage | S3 $0.023/GB-mo first 50TB, PUT $0.005/1K, GET $0.0004/1K, Egress $0.09/GB first 10TB over free tier
Log Aggregation for Task Logs
Tells you when things break and why. You cannot fix what you cannot see — this is how you keep downtime short.Pricing & free-tier limitsLoki OSS Free self-hosted | CloudWatch Free 5GB ingest + 5GB archive then $0.50/GB ingest + $0.03/GB-mo archive, $0.50/GB scan Logs Insights | Datadog Free 1GB/day trial then $0.10/GB ingest + $1.70 per 1M events indexed over 100, + $0.10/GB retention over
Monitoring for DAG Runs (Uptime, Duration)
Tells you when things break and why. You cannot fix what you cannot see — this is how you keep downtime short.Pricing & free-tier limitsPrometheus/Grafana OSS Free | Grafana Cloud Free 10K metrics + 50GB logs + 50GB traces then Pro $8/user/mo + $8 per 1K metrics over | AMP $0.90 per 1M ingested samples free 40M samples query, AMG $9/editor/mo | Datadog Infra Enterprise $23/host/mo + APM $34/host-mo + $0.20/1K spans overage
Alerting / Incident Management
Tells you when things break and why. You cannot fix what you cannot see — this is how you keep downtime short.Pricing & free-tier limitsSlack Webhook Free | SES Free 3K msgs/mo then $0.10 per 1K | Opsgenie Free 5 users + 1K SMS mins then Standard $29/user/mo + $0.30/SMS overage, Enterprise $71/user/mo | PagerDuty Free 1 user, Pro $41/user/mo + $0.25/SMS + $0.10/email over 500/mo
Secrets Management for Pipeline Credentials
A piece of your stack you may or may not need, depending on scope. Pick the option that fits — or skip it if your project doesn’t require this capability yet.Pricing & free-tier limitsSOPS/Env Free | Doppler Free 25 secrets 1 project then Team $18/user/mo 500 secrets + $0.05/secret over | AWS Secrets $0.40/secret/mo + $0.05 per 10K API calls, Free 30 days | HCP Vault $0.03/hr cluster ~$22/mo + $0.03 per 10K txns, Dedicated $0.48/hr ~$350/mo
CI/CD for DAG Deployment
Automates testing and deploying your code. Saves enormous time and prevents “works on my machine” releases.Pricing & free-tier limitsJenkins OSS Free | GitHub Actions Free 2000 Linux mins + 500MB storage then $0.008/min Linux + $0.25/GB-mo overage | GitLab Free 400 CI mins then Premium $29/user/mo + $0.008/min over | CircleCI Free 6000 credits then Scale $35/mo + $0.06/credit over, $0.10/GB egress over
Backup for Orchestrator Metadata DB
Copies of your data so a bug, hack, or outage doesn’t become permanent loss. Cheap insurance every serious project needs.Pricing & free-tier limitspg_dump Free + S3 storage $0.023/GB-mo | AWS Backup $0.05/GB-mo warm backup + $0.02/GB-mo warm + $0.03 restore over | RDS Snapshot Free up to DB size 7-day retention + $0.095/GB-mo over, PITR storage $0.095/GB-mo | Veeam $18 per instance/mo + $0.05/GB-mo overage
How this hobby pipeline orchestration checklist works.
Each requirement below is something a hobby pipeline orchestration build typically needs. Pick one of the four researched options — recommended, free, cheaper or paid — add your own with "Other", or skip the requirement if your project doesn't need it. Nothing is mandatory; the plan on the right tracks what you've decided so nothing gets forgotten.
Your picks are saved in this browser automatically, so you can come back anytime. Options are researched per build level and refreshed as vendors change their plans — always verify details on the provider's page before committing.