Engineering And Processing — hobby stack.
For each requirement below, pick the option that fits your build — recommended first, then free and cheaper alternatives — or skip what your project doesn't need. Tap the info icon next to any requirement to see why it matters.
Object Storage for Intermediate / Shuffle Data
Where you keep files users upload or you serve — images, videos, documents — and how fast they reach visitors around the world.Pricing & free-tier limitsFree: MinIO self-host $0, S3 Free Tier 5GB 12mo, 20k GET, 2k PUT. R2: 10GB free, 10M Class A ops free. Paid: S3 Standard $0.023/GB-mo (<50TB), $0.022/50-500TB, PUT $0.005/1k, GET $0.0004/1k. Express $0.16/GB-mo, $0.004/1k PUT. B2 $0.006/GB-mo, $0.005/1k PUT, egress $0.01/GB vs $0.09/GB AWS. Overage: R2 $0.015/GB-mo after free, egress $0.015/GB.
Batch Compute Cluster (Spark)
Where your code actually runs and serves requests. Picking the right host affects speed, scaling, and how much ops work you do.Pricing & free-tier limitsFree: Spark OSS $0 self-host. EMR Free: none. Paid: EMR $0.27/DPU-hr (1 DPU = 4vCPU 16GB) + EC2 m6i.xlarge $0.192/hr. Dataproc: $0.01/vCPU/hr + VM $0.0405/vCPU/hr, e2-standard-4 $0.134/hr, Spot -60% to -70%. Databricks: DBU $0.40/DBU (Jobs), Serverless $0.70/DBU, min 2 DBU. Overage: EMR 1-min min, Databricks Jobs $0.50/min over commit.
Workflow Orchestration & Scheduling
Scheduling, availability, and booking logic. Core to any appointment or reservation product.Pricing & free-tier limitsFree: Airflow/Prefect OSS $0 self-host, Dagster Cloud: 1 user, 30-day history free. Paid: MWAA small mw1.small $0.49/hr ~$353/mo + $0.10/GB env storage. Composer 2 small $0.65/hr ~$468/mo min. Astronomer Astro Team $750/mo includes 2 envs, $0.10/astro-unit hr. Dagster Starter $200/mo + $0.15/min compute overage. Overage: Scheduler $0.11/hr, worker auto-scale $0.15/hr.
Stream Processing Engine
A piece of your stack you may or may not need, depending on scope. Pick the option that fits — or skip it if your project doesn’t require this capability yet.Pricing & free-tier limitsFree: Flink/Kafka Streams OSS $0, Ververica trial 30 days. Paid: KDA Flink $0.11/KPU-hr (1 KPU=1vCPU 4GB), storage $0.023/GB-mo, durable $0.10/KPU-hr. Dataflow Streaming $0.069/vCPU-hr, 3.75GB $0.004/GB-hr. Confluent Flink: $0.15/CCU-hr + $0.12/GB processed. Overage: KDA checkpointing $0.023/GB-mo, Confluent min $0.50/hr.
Distributed Compute Framework (Ray/Dask)
Where your code actually runs and serves requests. Picking the right host affects speed, scaling, and how much ops work you do.Pricing & free-tier limitsFree: Ray/Dask OSS $0 self-host. Paid: Anyscale: $0 dev free tier (10 workers), then $1.00/hr per head + worker $0.50/hr + EC2. Coiled: Free 10k CPU-hr/mo trial, then Standard $0.10/CPU-hr + VM. Databricks Ray Jobs $0.45/DBU-hr. Overage: Anyscale $0.40/GB shuffle, Coiled $0.15/GB egress.
Data Warehouse for Processing Results Sink
A piece of your stack you may or may not need, depending on scope. Pick the option that fits — or skip it if your project doesn’t require this capability yet.Pricing & free-tier limitsFree: DuckDB local $0, BigQuery 10GB storage + 1TB query/mo free, Snowflake trial $400 credit 30d. Paid: BigQuery On-Demand $6.25/TB scanned, Storage $0.02/GB-mo active $0.01 long-term. BigQuery capacity: 100 slots $0.04/slot-hr ~$2000/mo for 100 slots. Snowflake: Compute $2.00/credit (XS=1cr/hr), Standard $2/credit, Enterprise $3/credit, storage $23/TB compressed. Overage: BQ autoscale slot idle $0, Snowflake min 60s, serverless $1.50/query.
Job Monitoring & Performance Observability
Tells you when things break and why. You cannot fix what you cannot see — this is how you keep downtime short.Pricing & free-tier limitsFree: Prometheus/Grafana OSS $0, Spark History Server free, Grafana Cloud: 10k series, 50GB logs, 50GB traces free tier. Datadog 14-day trial. Paid: Datadog APM $31/host/mo, custom metrics $0.05/100, logs $0.10/GB ingested, traces $0.80/GB. Grafana Cloud Pro $299/mo + $0.50/1000 series, $0.50/GB logs. Overage: Datadog $0.30/GB logs over 100GB, $0.12/1M traces.
Secrets Management
A piece of your stack you may or may not need, depending on scope. Pick the option that fits — or skip it if your project doesn’t require this capability yet.Pricing & free-tier limitsFree: Vault OSS $0 self-host, SSM Standard 10k params free. Paid: Secrets Manager $0.40/secret/mo + $0.05/10k API calls. SSM Advanced $0.05/param/mo. HCP Vault $0.32/hr (~$232/mo) small, plus $0.03/1k ops. Overage: Secrets Manager over 1M API calls $0.04/10k, Vault $0.15/hr large.
CI/CD for Data Pipelines
Automates testing and deploying your code. Saves enormous time and prevents “works on my machine” releases.Pricing & free-tier limitsFree: GitHub Actions 2000 min/mo free (Linux), 500MB storage free, self-hosted runners free. GitLab 400 min/mo free. Jenkins OSS $0. Paid: GitHub Team $4/user/mo, Enterprise $21/user/mo, hosted runner $0.008/min, 10x for 4-core. CircleCI $15/mo + 30k credits, $0.06/credit overage (~$0.008/min). Overage: GH Actions $0.008/min Linux, $0.016/min Windows.
Data Quality & Validation Checks
A piece of your stack you may or may not need, depending on scope. Pick the option that fits — or skip it if your project doesn’t require this capability yet.Pricing & free-tier limitsFree: Great Expectations OSS $0, Soda Core OSS free, Deequ $0. Paid: Great Expectations Cloud Team $500/mo includes 100 validation runs, $0.25 per run over. Soda Cloud $1000/mo 50 datasets, $15/dataset/mo over. Monte Carlo $30k/yr min + $0.30/validated GB. Overage: GX Cloud $0.50/validation run, Soda $0.12/1k checks.
Cost Tracking & FinOps for Jobs
Rates, labels, and tracking for physical goods. Needed once you actually ship products.Pricing & free-tier limitsFree: OpenCost/Kubecost Free cluster free, AWS Cost Explorer free, Vantage free tier 1k resources. Paid: Kubecost Enterprise $750/mo per cluster (50 nodes). Vantage $400/mo starts, $0.30/resource over 2500. CloudHealth $1k/mo min. Overage: Kubecost $12/node/mo over, Vantage $0.50/resource.
How this hobby engineering and processing checklist works.
Each requirement below is something a hobby engineering and processing build typically needs. Pick one of the four researched options — recommended, free, cheaper or paid — add your own with "Other", or skip the requirement if your project doesn't need it. Nothing is mandatory; the plan on the right tracks what you've decided so nothing gets forgotten.
Your picks are saved in this browser automatically, so you can come back anytime. Options are researched per build level and refreshed as vendors change their plans — always verify details on the provider's page before committing.