Engineering And Processing — small scale stack.
For each requirement below, pick the option that fits your build — recommended first, then free and cheaper alternatives — or skip what your project doesn't need. Tap the info icon next to any requirement to see why it matters.
Object Storage for Intermediate / Shuffle Data
Where you keep files users upload or you serve — images, videos, documents — and how fast they reach visitors around the world.Pricing & free-tier limitsFree: MinIO self-host $0, S3 Free Tier 5GB 12mo, 20k GET, 2k PUT. R2: 10GB free, 10M Class A ops free. Paid: S3 Standard $0.023/GB-mo (<50TB), $0.022/50-500TB, PUT $0.005/1k, GET $0.0004/1k. Express $0.16/GB-mo, $0.004/1k PUT. B2 $0.006/GB-mo, $0.005/1k PUT, egress $0.01/GB vs $0.09/GB AWS. Overage: R2 $0.015/GB-mo after free, egress $0.015/GB.
Batch Compute Cluster (Spark)
Where your code actually runs and serves requests. Picking the right host affects speed, scaling, and how much ops work you do.Pricing & free-tier limitsFree: Spark OSS $0 self-host. EMR Free: none. Paid: EMR $0.27/DPU-hr (1 DPU = 4vCPU 16GB) + EC2 m6i.xlarge $0.192/hr. Dataproc: $0.01/vCPU/hr + VM $0.0405/vCPU/hr, e2-standard-4 $0.134/hr, Spot -60% to -70%. Databricks: DBU $0.40/DBU (Jobs), Serverless $0.70/DBU, min 2 DBU. Overage: EMR 1-min min, Databricks Jobs $0.50/min over commit.
Workflow Orchestration & Scheduling
Scheduling, availability, and booking logic. Core to any appointment or reservation product.Pricing & free-tier limitsFree: Airflow/Prefect OSS $0 self-host, Dagster Cloud: 1 user, 30-day history free. Paid: MWAA small mw1.small $0.49/hr ~$353/mo + $0.10/GB env storage. Composer 2 small $0.65/hr ~$468/mo min. Astronomer Astro Team $750/mo includes 2 envs, $0.10/astro-unit hr. Dagster Starter $200/mo + $0.15/min compute overage. Overage: Scheduler $0.11/hr, worker auto-scale $0.15/hr.
Stream Processing Engine
A piece of your stack you may or may not need, depending on scope. Pick the option that fits — or skip it if your project doesn’t require this capability yet.Pricing & free-tier limitsFree: Flink/Kafka Streams OSS $0, Ververica trial 30 days. Paid: KDA Flink $0.11/KPU-hr (1 KPU=1vCPU 4GB), storage $0.023/GB-mo, durable $0.10/KPU-hr. Dataflow Streaming $0.069/vCPU-hr, 3.75GB $0.004/GB-hr. Confluent Flink: $0.15/CCU-hr + $0.12/GB processed. Overage: KDA checkpointing $0.023/GB-mo, Confluent min $0.50/hr.
Distributed Compute Framework (Ray/Dask)
Where your code actually runs and serves requests. Picking the right host affects speed, scaling, and how much ops work you do.Pricing & free-tier limitsFree: Ray/Dask OSS $0 self-host. Paid: Anyscale: $0 dev free tier (10 workers), then $1.00/hr per head + worker $0.50/hr + EC2. Coiled: Free 10k CPU-hr/mo trial, then Standard $0.10/CPU-hr + VM. Databricks Ray Jobs $0.45/DBU-hr. Overage: Anyscale $0.40/GB shuffle, Coiled $0.15/GB egress.
Data Warehouse for Processing Results Sink
A piece of your stack you may or may not need, depending on scope. Pick the option that fits — or skip it if your project doesn’t require this capability yet.Pricing & free-tier limitsFree: DuckDB local $0, BigQuery 10GB storage + 1TB query/mo free, Snowflake trial $400 credit 30d. Paid: BigQuery On-Demand $6.25/TB scanned, Storage $0.02/GB-mo active $0.01 long-term. BigQuery capacity: 100 slots $0.04/slot-hr ~$2000/mo for 100 slots. Snowflake: Compute $2.00/credit (XS=1cr/hr), Standard $2/credit, Enterprise $3/credit, storage $23/TB compressed. Overage: BQ autoscale slot idle $0, Snowflake min 60s, serverless $1.50/query.
Job Monitoring & Performance Observability
Tells you when things break and why. You cannot fix what you cannot see — this is how you keep downtime short.Pricing & free-tier limitsFree: Prometheus/Grafana OSS $0, Spark History Server free, Grafana Cloud: 10k series, 50GB logs, 50GB traces free tier. Datadog 14-day trial. Paid: Datadog APM $31/host/mo, custom metrics $0.05/100, logs $0.10/GB ingested, traces $0.80/GB. Grafana Cloud Pro $299/mo + $0.50/1000 series, $0.50/GB logs. Overage: Datadog $0.30/GB logs over 100GB, $0.12/1M traces.
Secrets Management
A piece of your stack you may or may not need, depending on scope. Pick the option that fits — or skip it if your project doesn’t require this capability yet.Pricing & free-tier limitsFree: Vault OSS $0 self-host, SSM Standard 10k params free. Paid: Secrets Manager $0.40/secret/mo + $0.05/10k API calls. SSM Advanced $0.05/param/mo. HCP Vault $0.32/hr (~$232/mo) small, plus $0.03/1k ops. Overage: Secrets Manager over 1M API calls $0.04/10k, Vault $0.15/hr large.
CI/CD for Data Pipelines
Automates testing and deploying your code. Saves enormous time and prevents “works on my machine” releases.Pricing & free-tier limitsFree: GitHub Actions 2000 min/mo free (Linux), 500MB storage free, self-hosted runners free. GitLab 400 min/mo free. Jenkins OSS $0. Paid: GitHub Team $4/user/mo, Enterprise $21/user/mo, hosted runner $0.008/min, 10x for 4-core. CircleCI $15/mo + 30k credits, $0.06/credit overage (~$0.008/min). Overage: GH Actions $0.008/min Linux, $0.016/min Windows.
Data Quality & Validation Checks
A piece of your stack you may or may not need, depending on scope. Pick the option that fits — or skip it if your project doesn’t require this capability yet.Pricing & free-tier limitsFree: Great Expectations OSS $0, Soda Core OSS free, Deequ $0. Paid: Great Expectations Cloud Team $500/mo includes 100 validation runs, $0.25 per run over. Soda Cloud $1000/mo 50 datasets, $15/dataset/mo over. Monte Carlo $30k/yr min + $0.30/validated GB. Overage: GX Cloud $0.50/validation run, Soda $0.12/1k checks.
Cost Tracking & FinOps for Jobs
Rates, labels, and tracking for physical goods. Needed once you actually ship products.Pricing & free-tier limitsFree: OpenCost/Kubecost Free cluster free, AWS Cost Explorer free, Vantage free tier 1k resources. Paid: Kubecost Enterprise $750/mo per cluster (50 nodes). Vantage $400/mo starts, $0.30/resource over 2500. CloudHealth $1k/mo min. Overage: Kubecost $12/node/mo over, Vantage $0.50/resource.
Message Queue / Event Ingestion
Runs slow work (emails, exports, payouts) in the background so users don’t wait. Keeps your app snappy.Pricing & free-tier limitsFree: Kafka/Redpanda OSS $0 self-host, Kinesis no free tier, Confluent $400 trial credit. Paid: Kinesis On-Demand $0.04/GB-hr ingress/egress + 1M writes $0.28. MSK Serverless $0.11/GB ingested, $0.12/GB egress. Confluent: $0.11/GB ingress, $0.12/GB egress, $0.10/GB storage/mo, CKU $0.50/hr. Upstash: Free 10k msgs/day, then $0.20/100k msgs. Overage: Kinesis $0.014/24M records extended retention.
Feature Store (Online/Offline)
A piece of your stack you may or may not need, depending on scope. Pick the option that fits — or skip it if your project doesn’t require this capability yet.Pricing & free-tier limitsFree: Feast OSS $0 self-host, Hopsworks Community free. Paid: SageMaker Feature Store $0.06/10k writes, $0.006/10k reads, storage $0.023/GB-mo online $0.21. Tecton: $0.10/feature/mo online serving, $2k/mo platform min + $0.15/million feature retrievals. Hopsworks Starter $450/mo includes 20 features online. Overage: Tecton batch $0.05/GB processed.
Metadata Catalog & Data Lineage
A piece of your stack you may or may not need, depending on scope. Pick the option that fits — or skip it if your project doesn’t require this capability yet.Pricing & free-tier limitsFree: DataHub/OpenMetadata/Marquez OSS $0 self-host, Atlan free 5 users 100 assets. Paid: Atlan Pro $1.5k/mo 10 users + 500 assets, $15/user/mo over. DataHub Cloud $0.15/MDUs hr ~$400/mo small. Alation $50k/yr min. Overage: Atlan $0.05/asset/mo over 1k.
Distributed Cache / Accelerated Shuffle Storage
Where you keep files users upload or you serve — images, videos, documents — and how fast they reach visitors around the world.Pricing & free-tier limitsFree: Redis OSS $0, Upstash Free 10k cmds/day, 256MB storage, Momento 50GB free. Paid: ElastiCache Serverless $0.125/GB-hr data stored $0.125 + $0.0035/10k ECPU, min $90/mo. Upstash $0.20/100k commands $0.38/GB-mo. Momento $0.50/GB storage $0.15/M ops. Overage: ElastiCache data $0.18/GB-hr burst.
Container Orchestration for Data Jobs
A piece of your stack you may or may not need, depending on scope. Pick the option that fits — or skip it if your project doesn’t require this capability yet.Pricing & free-tier limitsFree: k3s $0 self-host, EKS control plane $0.10/hr ~$72/mo, no free. Paid: EKS $0.10/hr cluster + EC2 workers m6i.xlarge $0.192/hr spot $0.07. GKE Autopilot $0.10/hr + $0.0445/vCPU/hr, $0.0049/GB-hr. Fargate $0.04048/vCPU/hr $0.004445/GB/hr, Spot -70%. Overage: EKS Fargate min 1 vCPU 2GB $0.05/hr.
Centralized Log Management
Tells you when things break and why. You cannot fix what you cannot see — this is how you keep downtime short.Pricing & free-tier limitsFree: Loki/OpenSearch OSS $0 self-host, Grafana Cloud Logs 50GB free, Axiom 500GB/mo free ingest. Better Stack 1GB/mo free. Paid: Axiom $25/mo 100GB + $0.25/GB over. Better Stack $24/mo 10GB, $0.30/GB over. Datadog Logs $0.10/GB ingested, $0.10/GB 7-day search, retention $1.70/1M log events. Loki Cloud Pro $100/mo + $0.50/GB. Overage: $0.35/GB over.
Backup & Disaster Recovery
Copies of your data so a bug, hack, or outage doesn’t become permanent loss. Cheap insurance every serious project needs.Pricing & free-tier limitsFree: Velero/Restic OSS $0 + storage cost only, AWS Backup free tier none. Paid: AWS Backup $0.05/GB-mo warm $0.01 cold, S3 CRR $0.02/GB transfer. Wasabi $6.99/TB/mo no egress, 1TB min. Druva $750/mo 1TB. Veeam $400/yr per 10 workloads. Overage: AWS Backup restore $0.02/GB, Wasabi over 1x egress $0.04/GB.
Managed Serverless Spark Platform
A piece of your stack you may or may not need, depending on scope. Pick the option that fits — or skip it if your project doesn’t require this capability yet.Pricing & free-tier limitsFree: EMR Serverless trial $200 credit, Dataproc trial $300 GCP credit. Paid: EMR Serverless $0.052624/vCPU-hr, $0.005778/GB-hr, $0.00576/storage GB-hr. Dataproc Serverless $0.06/vCPU-hr, $0.009/GB-hr. Databricks Serverless SQL $0.45/DBU-hr $0.70 serverless. Overage: EMR min 1 vCPU 1GB 1 min, Dataproc premium $0.40/hr.
How this small scale engineering and processing checklist works.
Each requirement below is something a small scale engineering and processing build typically needs. Pick one of the four researched options — recommended, free, cheaper or paid — add your own with "Other", or skip the requirement if your project doesn't need it. Nothing is mandatory; the plan on the right tracks what you've decided so nothing gets forgotten.
Your picks are saved in this browser automatically, so you can come back anytime. Options are researched per build level and refreshed as vendors change their plans — always verify details on the provider's page before committing.