Engineering And Processing — enterprise stack.
For each requirement below, pick the option that fits your build — recommended first, then free and cheaper alternatives — or skip what your project doesn't need. Tap the info icon next to any requirement to see why it matters.
Object Storage for Intermediate / Shuffle Data
Where you keep files users upload or you serve — images, videos, documents — and how fast they reach visitors around the world.Pricing & free-tier limitsFree: MinIO self-host $0, S3 Free Tier 5GB 12mo, 20k GET, 2k PUT. R2: 10GB free, 10M Class A ops free. Paid: S3 Standard $0.023/GB-mo (<50TB), $0.022/50-500TB, PUT $0.005/1k, GET $0.0004/1k. Express $0.16/GB-mo, $0.004/1k PUT. B2 $0.006/GB-mo, $0.005/1k PUT, egress $0.01/GB vs $0.09/GB AWS. Overage: R2 $0.015/GB-mo after free, egress $0.015/GB.
Batch Compute Cluster (Spark)
Where your code actually runs and serves requests. Picking the right host affects speed, scaling, and how much ops work you do.Pricing & free-tier limitsFree: Spark OSS $0 self-host. EMR Free: none. Paid: EMR $0.27/DPU-hr (1 DPU = 4vCPU 16GB) + EC2 m6i.xlarge $0.192/hr. Dataproc: $0.01/vCPU/hr + VM $0.0405/vCPU/hr, e2-standard-4 $0.134/hr, Spot -60% to -70%. Databricks: DBU $0.40/DBU (Jobs), Serverless $0.70/DBU, min 2 DBU. Overage: EMR 1-min min, Databricks Jobs $0.50/min over commit.
Workflow Orchestration & Scheduling
Scheduling, availability, and booking logic. Core to any appointment or reservation product.Pricing & free-tier limitsFree: Airflow/Prefect OSS $0 self-host, Dagster Cloud: 1 user, 30-day history free. Paid: MWAA small mw1.small $0.49/hr ~$353/mo + $0.10/GB env storage. Composer 2 small $0.65/hr ~$468/mo min. Astronomer Astro Team $750/mo includes 2 envs, $0.10/astro-unit hr. Dagster Starter $200/mo + $0.15/min compute overage. Overage: Scheduler $0.11/hr, worker auto-scale $0.15/hr.
Stream Processing Engine
A piece of your stack you may or may not need, depending on scope. Pick the option that fits — or skip it if your project doesn’t require this capability yet.Pricing & free-tier limitsFree: Flink/Kafka Streams OSS $0, Ververica trial 30 days. Paid: KDA Flink $0.11/KPU-hr (1 KPU=1vCPU 4GB), storage $0.023/GB-mo, durable $0.10/KPU-hr. Dataflow Streaming $0.069/vCPU-hr, 3.75GB $0.004/GB-hr. Confluent Flink: $0.15/CCU-hr + $0.12/GB processed. Overage: KDA checkpointing $0.023/GB-mo, Confluent min $0.50/hr.
Distributed Compute Framework (Ray/Dask)
Where your code actually runs and serves requests. Picking the right host affects speed, scaling, and how much ops work you do.Pricing & free-tier limitsFree: Ray/Dask OSS $0 self-host. Paid: Anyscale: $0 dev free tier (10 workers), then $1.00/hr per head + worker $0.50/hr + EC2. Coiled: Free 10k CPU-hr/mo trial, then Standard $0.10/CPU-hr + VM. Databricks Ray Jobs $0.45/DBU-hr. Overage: Anyscale $0.40/GB shuffle, Coiled $0.15/GB egress.
Data Warehouse for Processing Results Sink
A piece of your stack you may or may not need, depending on scope. Pick the option that fits — or skip it if your project doesn’t require this capability yet.Pricing & free-tier limitsFree: DuckDB local $0, BigQuery 10GB storage + 1TB query/mo free, Snowflake trial $400 credit 30d. Paid: BigQuery On-Demand $6.25/TB scanned, Storage $0.02/GB-mo active $0.01 long-term. BigQuery capacity: 100 slots $0.04/slot-hr ~$2000/mo for 100 slots. Snowflake: Compute $2.00/credit (XS=1cr/hr), Standard $2/credit, Enterprise $3/credit, storage $23/TB compressed. Overage: BQ autoscale slot idle $0, Snowflake min 60s, serverless $1.50/query.
Job Monitoring & Performance Observability
Tells you when things break and why. You cannot fix what you cannot see — this is how you keep downtime short.Pricing & free-tier limitsFree: Prometheus/Grafana OSS $0, Spark History Server free, Grafana Cloud: 10k series, 50GB logs, 50GB traces free tier. Datadog 14-day trial. Paid: Datadog APM $31/host/mo, custom metrics $0.05/100, logs $0.10/GB ingested, traces $0.80/GB. Grafana Cloud Pro $299/mo + $0.50/1000 series, $0.50/GB logs. Overage: Datadog $0.30/GB logs over 100GB, $0.12/1M traces.
Secrets Management
A piece of your stack you may or may not need, depending on scope. Pick the option that fits — or skip it if your project doesn’t require this capability yet.Pricing & free-tier limitsFree: Vault OSS $0 self-host, SSM Standard 10k params free. Paid: Secrets Manager $0.40/secret/mo + $0.05/10k API calls. SSM Advanced $0.05/param/mo. HCP Vault $0.32/hr (~$232/mo) small, plus $0.03/1k ops. Overage: Secrets Manager over 1M API calls $0.04/10k, Vault $0.15/hr large.
CI/CD for Data Pipelines
Automates testing and deploying your code. Saves enormous time and prevents “works on my machine” releases.Pricing & free-tier limitsFree: GitHub Actions 2000 min/mo free (Linux), 500MB storage free, self-hosted runners free. GitLab 400 min/mo free. Jenkins OSS $0. Paid: GitHub Team $4/user/mo, Enterprise $21/user/mo, hosted runner $0.008/min, 10x for 4-core. CircleCI $15/mo + 30k credits, $0.06/credit overage (~$0.008/min). Overage: GH Actions $0.008/min Linux, $0.016/min Windows.
Data Quality & Validation Checks
A piece of your stack you may or may not need, depending on scope. Pick the option that fits — or skip it if your project doesn’t require this capability yet.Pricing & free-tier limitsFree: Great Expectations OSS $0, Soda Core OSS free, Deequ $0. Paid: Great Expectations Cloud Team $500/mo includes 100 validation runs, $0.25 per run over. Soda Cloud $1000/mo 50 datasets, $15/dataset/mo over. Monte Carlo $30k/yr min + $0.30/validated GB. Overage: GX Cloud $0.50/validation run, Soda $0.12/1k checks.
Cost Tracking & FinOps for Jobs
Rates, labels, and tracking for physical goods. Needed once you actually ship products.Pricing & free-tier limitsFree: OpenCost/Kubecost Free cluster free, AWS Cost Explorer free, Vantage free tier 1k resources. Paid: Kubecost Enterprise $750/mo per cluster (50 nodes). Vantage $400/mo starts, $0.30/resource over 2500. CloudHealth $1k/mo min. Overage: Kubecost $12/node/mo over, Vantage $0.50/resource.
Message Queue / Event Ingestion
Runs slow work (emails, exports, payouts) in the background so users don’t wait. Keeps your app snappy.Pricing & free-tier limitsFree: Kafka/Redpanda OSS $0 self-host, Kinesis no free tier, Confluent $400 trial credit. Paid: Kinesis On-Demand $0.04/GB-hr ingress/egress + 1M writes $0.28. MSK Serverless $0.11/GB ingested, $0.12/GB egress. Confluent: $0.11/GB ingress, $0.12/GB egress, $0.10/GB storage/mo, CKU $0.50/hr. Upstash: Free 10k msgs/day, then $0.20/100k msgs. Overage: Kinesis $0.014/24M records extended retention.
Feature Store (Online/Offline)
A piece of your stack you may or may not need, depending on scope. Pick the option that fits — or skip it if your project doesn’t require this capability yet.Pricing & free-tier limitsFree: Feast OSS $0 self-host, Hopsworks Community free. Paid: SageMaker Feature Store $0.06/10k writes, $0.006/10k reads, storage $0.023/GB-mo online $0.21. Tecton: $0.10/feature/mo online serving, $2k/mo platform min + $0.15/million feature retrievals. Hopsworks Starter $450/mo includes 20 features online. Overage: Tecton batch $0.05/GB processed.
Metadata Catalog & Data Lineage
A piece of your stack you may or may not need, depending on scope. Pick the option that fits — or skip it if your project doesn’t require this capability yet.Pricing & free-tier limitsFree: DataHub/OpenMetadata/Marquez OSS $0 self-host, Atlan free 5 users 100 assets. Paid: Atlan Pro $1.5k/mo 10 users + 500 assets, $15/user/mo over. DataHub Cloud $0.15/MDUs hr ~$400/mo small. Alation $50k/yr min. Overage: Atlan $0.05/asset/mo over 1k.
Distributed Cache / Accelerated Shuffle Storage
Where you keep files users upload or you serve — images, videos, documents — and how fast they reach visitors around the world.Pricing & free-tier limitsFree: Redis OSS $0, Upstash Free 10k cmds/day, 256MB storage, Momento 50GB free. Paid: ElastiCache Serverless $0.125/GB-hr data stored $0.125 + $0.0035/10k ECPU, min $90/mo. Upstash $0.20/100k commands $0.38/GB-mo. Momento $0.50/GB storage $0.15/M ops. Overage: ElastiCache data $0.18/GB-hr burst.
Container Orchestration for Data Jobs
A piece of your stack you may or may not need, depending on scope. Pick the option that fits — or skip it if your project doesn’t require this capability yet.Pricing & free-tier limitsFree: k3s $0 self-host, EKS control plane $0.10/hr ~$72/mo, no free. Paid: EKS $0.10/hr cluster + EC2 workers m6i.xlarge $0.192/hr spot $0.07. GKE Autopilot $0.10/hr + $0.0445/vCPU/hr, $0.0049/GB-hr. Fargate $0.04048/vCPU/hr $0.004445/GB/hr, Spot -70%. Overage: EKS Fargate min 1 vCPU 2GB $0.05/hr.
Centralized Log Management
Tells you when things break and why. You cannot fix what you cannot see — this is how you keep downtime short.Pricing & free-tier limitsFree: Loki/OpenSearch OSS $0 self-host, Grafana Cloud Logs 50GB free, Axiom 500GB/mo free ingest. Better Stack 1GB/mo free. Paid: Axiom $25/mo 100GB + $0.25/GB over. Better Stack $24/mo 10GB, $0.30/GB over. Datadog Logs $0.10/GB ingested, $0.10/GB 7-day search, retention $1.70/1M log events. Loki Cloud Pro $100/mo + $0.50/GB. Overage: $0.35/GB over.
Backup & Disaster Recovery
Copies of your data so a bug, hack, or outage doesn’t become permanent loss. Cheap insurance every serious project needs.Pricing & free-tier limitsFree: Velero/Restic OSS $0 + storage cost only, AWS Backup free tier none. Paid: AWS Backup $0.05/GB-mo warm $0.01 cold, S3 CRR $0.02/GB transfer. Wasabi $6.99/TB/mo no egress, 1TB min. Druva $750/mo 1TB. Veeam $400/yr per 10 workloads. Overage: AWS Backup restore $0.02/GB, Wasabi over 1x egress $0.04/GB.
Managed Serverless Spark Platform
A piece of your stack you may or may not need, depending on scope. Pick the option that fits — or skip it if your project doesn’t require this capability yet.Pricing & free-tier limitsFree: EMR Serverless trial $200 credit, Dataproc trial $300 GCP credit. Paid: EMR Serverless $0.052624/vCPU-hr, $0.005778/GB-hr, $0.00576/storage GB-hr. Dataproc Serverless $0.06/vCPU-hr, $0.009/GB-hr. Databricks Serverless SQL $0.45/DBU-hr $0.70 serverless. Overage: EMR min 1 vCPU 1GB 1 min, Dataproc premium $0.40/hr.
GPU-Accelerated Processing Cluster
Powers AI features — model access, embeddings, and inference. Costs scale with usage, so watch the meter.Pricing & free-tier limitsFree: Own hardware $0 beyond power. Paid: g5.xlarge (A10G) $1.006/hr, p4d.24xlarge (8x A100) $32.77/hr (~$4.10/GPU-hr), A100 40GB $1.50/hr spot $0.80/hr. Lambda Labs A100 $1.29/hr, H100 $2.49/hr. GCP A2 A100 $1.88/hr per GPU, A3 H100 $3.22/hr per GPU. RunPod $0.69/hr A100 SXM, spot $0.49. Overage: Inter-GPU egress $0.01/GB, H100 reservation $4.50/hr.
Audit Logs & Compliance Event Store
Lets external services notify you of events in real time. Core to integrating payments, KYC, and third-party workflows.Pricing & free-tier limitsFree: CloudTrail 1 trail free (90d), Auditbeat/Wazuh OSS $0, CloudTrail Lake 30d. Paid: CloudTrail Lake $0.75/GB ingested + $0.023/GB-mo storage 7yr. Splunk $0.60/GB ingested min 1GB/day $675/mo. Datadog Audit $0.30/GB ingested $0.10/GB over 30d retained. Overage: CloudTrail additional trails $2.00/100k events.
Secrets Rotation & HSM Integration
A piece of your stack you may or may not need, depending on scope. Pick the option that fits — or skip it if your project doesn’t require this capability yet.Pricing & free-tier limitsFree: Vault OSS + SoftHSM $0, Infisical Free 1k secrets 5 members. Paid: CloudHSM $2.23/hr per HSM (~$1627/mo) plus $0.50/hr per AZ. HCP Vault Enterprise $1.10/hr + $0.10/1k ops. Dopplar Team $6/user/mo $18/mo 1k secrets, over $0.10/secret. Overage: CloudHSM backup $0.03/GB-mo.
Auto-Scaling & Spot Cluster Manager
A piece of your stack you may or may not need, depending on scope. Pick the option that fits — or skip it if your project doesn’t require this capability yet.Pricing & free-tier limitsFree: Karpenter/CA OSS $0 self-host, Cast AI free tier $200/mo savings share no charge. Paid: Spot.io Ocean $0 + 20% savings share, avg effective $0.05/node-hr platform fee, Enterprise $750/mo. Cast AI $250/mo cluster + 20% savings fee. Overage: Spot instance interruption handling free, on-demand fallback billed at $0.192/hr m6i.xlarge.
Data Lake Table Format Manager (Iceberg/Delta/Hudi)
A piece of your stack you may or may not need, depending on scope. Pick the option that fits — or skip it if your project doesn’t require this capability yet.Pricing & free-tier limitsFree: Iceberg/Delta/Nessie OSS $0 self-host. Paid: S3 Tables $0.023/GB-mo storage + $0.005/10k requests + $0.025/GB compaction. Tabular $350/mo dev, $0.10/GB scanned + $2k/mo prod min. LakeFS Cloud Free 100GB/mo, then $100/mo 1TB. Dremio Arctic Free 1k tables, then $0.06/GB queried. Overage: S3 Tables compaction $0.025/GB processed.
Distributed SQL Query Engine (Presto/Trino/Athena)
Persistent storage for your app’s data — users, products, orders. The single most important architectural decision for most projects.Pricing & free-tier limitsFree: Trino/Presto OSS $0, Athena no free tier beyond $0.00 first TB? Actually 1TB query free first year? Paid: Athena v3 $5.00/TB scanned (min 10MB/query), provisioned capacity 96 DPU $2.50/hr. Trino on EKS self-host $0 + EC2. Starburst Galaxy: Free 10 clusters $50 credit, then $0.10/query + $0.45/DPU-hr, Enterprise $2.5k/mo min. Dremio $0.06/CU-min (~$50/TB scanned). Overage: Athena scanned $0.005/GB over free 1TB trial.
Change Data Capture (CDC) Infrastructure
A piece of your stack you may or may not need, depending on scope. Pick the option that fits — or skip it if your project doesn’t require this capability yet.Pricing & free-tier limitsFree: Debezium/Maxwell OSS $0 self-host, Airbyte OSS free 14-day. Paid: AWS DMS Serverless 10 CDC capacity free? No free; $0.36/Capacity hr (2 vCPU 8GB) + storage $0.12/GB-mo. Fivetran CDC free 14d then $500/mo 1.5M monthly active rows, $1.20/1k MAR over. Striim $1.5k/mo node, $0.20/GB processed. HVR $10k/yr min. Overage: DMS replication $0.025/GB transfer.
Schema Registry
A piece of your stack you may or may not need, depending on scope. Pick the option that fits — or skip it if your project doesn’t require this capability yet.Pricing & free-tier limitsFree: Apicurio/Glue Schema Registry free tier? Glue SR free operations 1M checks free/mo, Apicurio OSS $0. Paid: Confluent SR Essentials $0 + 1000 schemas free, then $0.25/100k ops. Advanced Gov $1.00/GB data governed + $0.10/1k schema ops, $500/mo min. AWS Glue SR free + S3. Overage: Confluent 1000+ schemas $0.01/schema/day.
Data Masking & PII Tokenization Service
A piece of your stack you may or may not need, depending on scope. Pick the option that fits — or skip it if your project doesn’t require this capability yet.Pricing & free-tier limitsFree: Presidio OSS $0 self-host, Macie 30-day trial. Paid: Macie $1.00/GB scanned first 50TB, $0.50 50-500TB, classification $0.10/GB job. Immuta $40k/yr min + $0.05/1k queries masked. Privacera $2k/mo starter. BigID $50k/yr. Overage: Macie over 1PB $0.25/GB.
Artifact Registry & Environment Management
Stores API keys and credentials safely, separate from your code. Leaking secrets is one of the most common breaches.Pricing & free-tier limitsFree: Harbor/MLflow OSS $0, ECR 500MB storage free tier 12mo, GHCR 500MB free + 1k build min. Paid: ECR $0.10/GB-mo storage, $0.30/GB transfer internet, free intra-VPC. GHCR Team $0.008/GB storage, $0.50/GB transfer over. Artifactory Cloud Pro $57/mo 10GB, $0.65/GB over, $2k/mo Enterprise. Overage: ECR batch scan $0.03/scan.
Distributed Tracing for Data Pipelines
Tells you when things break and why. You cannot fix what you cannot see — this is how you keep downtime short.Pricing & free-tier limitsFree: OTEL/Jaeger/Tempo OSS $0 self-host, Grafana Cloud 50GB traces free, Honeycomb Free 20M events/mo. Paid: Grafana Cloud Traces $0.50/GB ingested over 50GB, $8/1000 retention-hr. Datadog APM $35/host/mo + $0.22/1M spans ingested over 1M. Honeycomb Pro $100/mo + $0.10/1M events over 20M. New Relic 100GB free, then $0.25/GB. Overage: Jaeger S3 $0.023/GB-mo storage.
Private Networking & VPC for Data Processing
A piece of your stack you may or may not need, depending on scope. Pick the option that fits — or skip it if your project doesn’t require this capability yet.Pricing & free-tier limitsFree: WireGuard OSS $0, Tailscale Free 100 devices 1 user free, Cloudflare Tunnel free. Paid: AWS PrivateLink $0.01/hr endpoint + $0.01/GB data processed, TGW $0.05/hr attachment + $0.02/GB processed. Direct Connect 1Gbps dedicated $0.30/hr (~$216/mo) + $0.02/GB egress from AWS, hosted $0.30/hr. Tailscale Enterprise $18/user/mo. Zscaler $25/user/mo. Overage: PrivateLink $0.015/GB over 1TB.
Cross-Region Replication & Active-Active Fabric
Runs in multiple regions for speed and resilience. Enterprise-grade reliability and the hardest to retrofit.Pricing & free-tier limitsFree: MM2/rclone OSS $0 + transfer cost. Paid: S3 CRR $0.02/GB inter-region transfer + $0.015/1k replication request, storage double. MSK replication: EC2 for MM2 $0.192/hr. Confluent Cluster Linking $0.18/GB replicatd, CRR dedicated $0.50/hr per link. Striim multi-DC $3k/mo per cluster. Overage: Inter-region $0.02/GB US->EU $0.02, EU->AP $0.03.
Policy Enforcement & Governance (OPA/Lake Formation)
A piece of your stack you may or may not need, depending on scope. Pick the option that fits — or skip it if your project doesn’t require this capability yet.Pricing & free-tier limitsFree: OPA/Kyverno OSS $0 self-host, Styra DAS Free 10 policies 1 cluster free. Lake Formation free, $5/100k requests free tier. Paid: Styra Enterprise $1000/mo + $0.20/decision 1M over, Lake Formation tags $0.20/100k requests over free. Immuta Enterprise policy $40k/yr. Privacera Enterprise $3k/mo. Overage: Styra $0.15/100k decisions.
High-Throughput Interconnect & Local NVMe Shuffle
A piece of your stack you may or may not need, depending on scope. Pick the option that fits — or skip it if your project doesn’t require this capability yet.Pricing & free-tier limitsFree: Instance store NVMe included with i4i/d3en (self-managed). Paid: EBS gp3 $0.08/GB-mo, io2 $0.125/GB-mo + $0.065/1000 IOPS, io2 Block Express $0.125/GB + $0.015/1k IOPS. FSx Lustre $0.14/GB-mo SSD $0.0012/GB-hr + 40MB/TiB baseline throughput $0.20. EFA free with instances, but p4d $32/hr. Local SSD GCP: Local SSD $0.11/GB-mo persistent, $0.108/instance ephemeral. Overage: NVMe over-provision $0.15/GB-mo.
Incident Management & Job SLA Alerting
Tells you when things break and why. You cannot fix what you cannot see — this is how you keep downtime short.Pricing & free-tier limitsFree: Grafana OnCall OSS $0 self-host, PagerDuty Free 1 user 100 notifs/mo, Opsgenie Free 5 users 100 alerts. Paid: PagerDuty Pro $29/user/mo (bills monthly) $0.10 SMS over 300, Enterprise $39/user/mo + $20/user/mo for event intelligence. Opsgenie $11/user/mo Stakeholder $29/user/mo $0.10/min call. Rootly $17/user/mo. Overage: PagerDuty phone $0.15/call, SMS $0.10.
Compliance Archival & WORM Immutable Storage
Where you keep files users upload or you serve — images, videos, documents — and how fast they reach visitors around the world.Pricing & free-tier limitsFree: MinIO WORM $0 self-host + disk cost, Object Lock free feature. Paid: S3 Glacier Instant $0.004/GB-mo, Flexible $0.0036/GB-mo $0.03/1k puts, Deep Archive $0.00099/GB-mo ($0.99/TB-mo) egress $0.09/GB. Object Lock no extra. Wasabi compliance $6.99/TB-mo min 90d retention. B2 Object Lock same $0.006/GB-mo. Overage: Glacier restore $0.025/GB bulk, $0.10/GB standard expedited $0.30/GB.
KMS & Customer-Managed Keys + Confidential Compute
Where your code actually runs and serves requests. Picking the right host affects speed, scaling, and how much ops work you do.Pricing & free-tier limitsFree: Vault Transit $0 self-host, KMS free 20k requests/mo AWS. Paid: AWS KMS $1.00/key/mo + $0.03/10k requests, $0.03/10k asymmetric. CloudHSM $2.23/hr per HSM ~$1627/mo. GCP External KMS $1.50/key version/mo + $0.03/10k ops. Confidential VMs AMD SEV premium 10% on m6i (~$0.02/hr extra). Nitro Enclaves free, attestation $0. Nitro $0. Overage: KMS over 100k req $0.02/10k.
Multi-Cloud / Hybrid Data Processing Fabric
A piece of your stack you may or may not need, depending on scope. Pick the option that fits — or skip it if your project doesn’t require this capability yet.Pricing & free-tier limitsFree: Terraform OSS $0, Alluxio OSS $0 self-host, Pulumi Free 1 member 100 resources. Spacelift Free 1 user. Paid: Terraform Cloud Team $70/mo 5 users + $0.00014/resource-hr over, Enterprise $150k/yr min. Alluxio Enterprise $12k/mo per TB cache $0.30/GB-hr managed. Pulumi Team $80/mo. Spacelift $200/mo 5 stacks. Overage: Multi-cloud egress $0.08-0.12/GB (AWS->GCP $0.09/GB, GCP->AWS $0.12/GB).
How this enterprise engineering and processing checklist works.
Each requirement below is something a enterprise engineering and processing build typically needs. Pick one of the four researched options — recommended, free, cheaper or paid — add your own with "Other", or skip the requirement if your project doesn't need it. Nothing is mandatory; the plan on the right tracks what you've decided so nothing gets forgotten.
Your picks are saved in this browser automatically, so you can come back anytime. Options are researched per build level and refreshed as vendors change their plans — always verify details on the provider's page before committing.