Integration And ETL ELT — enterprise stack.
For each requirement below, pick the option that fits your build — recommended first, then free and cheaper alternatives — or skip what your project doesn't need. Tap the info icon next to any requirement to see why it matters.
ETL/ELT Platform
A piece of your stack you may or may not need, depending on scope. Pick the option that fits — or skip it if your project doesn’t require this capability yet.Pricing & free-tier limitsAirbyte OSS Free unlimited connectors self-hosted; Cloud Free: 14-day trial + 1M credits/mo dev free. $2.50/credit after (1 credit=1k rows synced). Fivetran $0.10/credit, 500k MAR/mo free for 14d. Hevo $200/mo for 15M events. Overage $0.0002/record (~$0.20/1k)
Connector Marketplace / Pre-built Connectors
A piece of your stack you may or may not need, depending on scope. Pick the option that fits — or skip it if your project doesn’t require this capability yet.Pricing & free-tier limitsAirbyte OSS 600+ free marketplace access self-host; Cloud marketplace included in credit. Singer OSS free 200+ taps. Fivetran connector pack $120/mo per connector MAR based, MAR overage $0.10/credit (~1.2M rows=$120). Estuary Free 100GB/mo ingress, Pro $0.15/GB after.
Transformation Engine (T Layer)
A piece of your stack you may or may not need, depending on scope. Pick the option that fits — or skip it if your project doesn’t require this capability yet.Pricing & free-tier limitsdbt Core free unlimited models self-hosted. dbt Cloud Free: 1 dev seat + 1 prod job, Team $100/mo/seat includes 1k model runs. Overage $0.20/model run. Dataform GCP free within BigQuery quotas. Coalesce $2k/mo starter 5 seats.
Data Pipeline Queue / Message Bus
Runs slow work (emails, exports, payouts) in the background so users don’t wait. Keeps your app snappy.Pricing & free-tier limitsKafka OSS free self-hosted. Confluent Cloud Free $0/mo, $400 trial credit + 10GB ingress free. Standard $0.11/GB ingress + $0.21/hr per CKU. AWS SQS Free 1M req/mo, $0.40/1M after. MSK $0.21/hr per broker m5.large + $0.10/GB storage.
Batch Processing Engine
A piece of your stack you may or may not need, depending on scope. Pick the option that fits — or skip it if your project doesn’t require this capability yet.Pricing & free-tier limitsSpark OSS free. Glue Free 1M DPU-seconds trial month, then $0.44/DPU-hr (1 DPU=4vCPU 16GB). Databricks $0.40/DBU + infra $0.10/DBU. Dataflow Batch $0.056/vCPU-hr, free 25 hr/mo. EMR Serverless $0.27/hr m5.xlarge + EC2.
Object Storage for Staging / Landing Zone
Where you keep files users upload or you serve — images, videos, documents — and how fast they reach visitors around the world.Pricing & free-tier limitsMinIO free self-host unlimited. S3 Standard Free 5GB for 12mo, after $0.023/GB/mo, PUT $0.005/1k, GET $0.0004/1k. B2 $0.006/GB ($6/TB/mo), egress $0.01/GB first 10GB/day free. R2 Free 10GB/mo storage, $0.015/GB after. Overage retrieval $0.01/GB.
Orchestration for Integration Jobs
A piece of your stack you may or may not need, depending on scope. Pick the option that fits — or skip it if your project doesn’t require this capability yet.Pricing & free-tier limitsAirflow OSS free unlimited DAGs. Astro Free dev deployment 14-day trial $0, Team $30/day (~$900/mo) includes 5 workers + 20 task runs/day. MWAA $0.50/hr env + $0.10/worker-hr. Prefect Cloud Free 20k task runs/mo, $0.002/task after, Team $200/mo.
Secrets Management for Connector Creds
A piece of your stack you may or may not need, depending on scope. Pick the option that fits — or skip it if your project doesn’t require this capability yet.Pricing & free-tier limitsVault OSS free self-host. HCP Vault Free 25 secrets, after $0.50/secret/mo. AWS Secrets Manager Free 30d trial, $0.40/secret/mo + 10k API calls $0.05. Doppler Free 100 secrets, Starter $18/user/mo. Infisical Free 500 secrets, $5/user/mo Pro after.
Monitoring for Integration / Data Observability
Tells you when things break and why. You cannot fix what you cannot see — this is how you keep downtime short.Pricing & free-tier limitsGrafana OSS free. Datadog Free 1-day retention up to 5 hosts. Pro $15/host/mo + logs $0.10/GB ingested first 1GB/day free. Monte Carlo Starter $20k/yr 100 tables, $150/table extra. Overage $0.30/1k pipeline events after 100k/mo free.
Audit Logs (WORM Immutable)
Controls who can do what inside your admin tooling. Limits blast radius if an account is compromised.Pricing & free-tier limitsS3 Object Lock free feature - storage $0.023/GB. CloudTrail Lake Free 100k events/mo 3mo promo, $2.50/GB ingested after. CloudWatch Logs Free 5GB/mo, $0.50/GB after retention 7yr WORM $0.01/GB/mo. Splunk Cloud $1.80/GB/day ingest min ~$108/mo.
Backup for Pipeline State / Checkpointing
Copies of your data so a bug, hack, or outage doesn’t become permanent loss. Cheap insurance every serious project needs.Pricing & free-tier limitsVelero free OSS. AWS Backup Free 50GB warm first month, $0.05/GB/mo warm, $0.01/GB cold. S3 Versioning storage at $0.023/GB. Restic free + B2 $6/TB/mo. Veeam $0.50/workload/day min $250/mo. Cross-region transfer $0.02/GB.
CDC Tool (Change Data Capture)
A piece of your stack you may or may not need, depending on scope. Pick the option that fits — or skip it if your project doesn’t require this capability yet.Pricing & free-tier limitsDebezium OSS free unlimited connectors self-hosted. Confluent Debezium Premium Connector $0.22/connector-hr. Striim Free dev 1 connector, Prod $3,500/mo per server, $0.001 per change event after 10M included. Fivetran HVR $1,200/mo per source.
Stream Processing Engine
A piece of your stack you may or may not need, depending on scope. Pick the option that fits — or skip it if your project doesn’t require this capability yet.Pricing & free-tier limitsFlink OSS free self-host. Confluent Flink Free up to 10 CU trial, $0.11/CU-hr Standard. Ververica Cloud Free $300 trial credit, Standard $0.60/CU-hr (1 CU=1vCPU 4GB). Kinesis Analytics $0.11/KPU/hr. State storage $0.10/GB/mo.
Schema Registry & Evolution Management
A piece of your stack you may or may not need, depending on scope. Pick the option that fits — or skip it if your project doesn’t require this capability yet.Pricing & free-tier limitsApicurio OSS free unlimited schemas. Confluent Cloud Free 100 schemas. Essentials $1/schema/mo after. Glue Schema Registry Free 1M validations/mo, $1/1M after no per-schema charge. Enterprise $2.50/schema/mo with evolution rules + compatibility.
Data Quality & Validation Framework
A piece of your stack you may or may not need, depending on scope. Pick the option that fits — or skip it if your project doesn’t require this capability yet.Pricing & free-tier limitsGE OSS free unlimited tests. Soda Cloud Free 10k checks/mo, Team $300/mo 50k checks $0.006/check. Metaplane Free 1k tests/mo. Monte Carlo $30k/yr starter 100 tables $150/table extra, $0.05/check overage.
Data Warehouse / Lakehouse Destination
A piece of your stack you may or may not need, depending on scope. Pick the option that fits — or skip it if your project doesn’t require this capability yet.Pricing & free-tier limitsDuckDB free OSS; MotherDuck Free 10GB storage $0. BigQuery Free 10GB storage + 1TB queries/mo, $0.02/GB storage after $6.25/TB query. Snowflake Standard $2/credit compute storage $23/TB/mo free 30d $400 credits. ClickHouse Cloud Free 1GB storage.
Rate Limiting & Throttling for Source APIs
Controls and protects your APIs — quotas, abuse prevention, and firewalls. Important once you have real traffic or many clients.Pricing & free-tier limitsNGINX OSS free. AWS API Gateway Free 1M calls/mo 12mo, REST $3.50/1M, HTTP $1.00/1M. Cloudflare Workers Free 100k req/day, Pro $5/mo includes 10M req $0.50/M after. Kong Konnect Free 10k req/min, Team $250/mo after. Overage throttled requests free.
API Gateway for Data APIs
Controls and protects your APIs — quotas, abuse prevention, and firewalls. Important once you have real traffic or many clients.Pricing & free-tier limitsKong OSS free unlimited routes. AWS API Gateway HTTP $1/M req after free 1M, REST $3.50/M, WebSocket $1/M + $0.25/conn-hr. Apigee X Free trial $300 credit, $0.02/10k calls paygo. Cloudflare API Gateway included with Pro/Enterprise.
Cost Management & Credit Consumption Monitoring
Tells you when things break and why. You cannot fix what you cannot see — this is how you keep downtime short.Pricing & free-tier limitsOpenCost free OSS unlimited. Infracost OSS free. CloudZero Free trial 30d, Starter $350/mo for $20k spend tracked. Datadog CCM free if Datadog customer standalone $0.01/resource-hr. CloudHealth $15k/yr minimum. Overage tracking $1.50/1k credits.
Reverse ETL Tool
A piece of your stack you may or may not need, depending on scope. Pick the option that fits — or skip it if your project doesn’t require this capability yet.Pricing & free-tier limitsRudderStack Free 1M events/mo, $0.10/1k after. Census Free 10 dest 1 model, Team $800/mo 10k syncs. Hightouch Starter $300/mo 5 syncs 500k rows, $0.15/1k rows overage. Enterprise $3k+/mo unlimited.
Lineage Tracking
Rates, labels, and tracking for physical goods. Needed once you actually ship products.Pricing & free-tier limitsOpenLineage + Marquez free OSS unlimited events. DataHub Cloud Free 1M lineage events/mo, Standard $400/mo 100 tables lineage $0.40/table after. Atlan Starter $2k/mo 10 users $200/user extra. Alation $1.2k/user/yr min 20 users.
Metadata Catalog for Integrations
A piece of your stack you may or may not need, depending on scope. Pick the option that fits — or skip it if your project doesn’t require this capability yet.Pricing & free-tier limitsDataHub OSS free unlimited metadata. Amundsen free. Atlan Free trial 14d. DataHub Cloud Trial $0 14d, Standard $400/mo 100 assets $0.40/asset after. Alation $90k/yr minimum. Collibra $100k/yr start.
Dead Letter Queue & Error Handling
Runs slow work (emails, exports, payouts) in the background so users don’t wait. Keeps your app snappy.Pricing & free-tier limitsSQS DLQ feature free, SQS Free 1M req/mo $0.40/1M after. RabbitMQ OSS free. Confluent DLQ included in topic cost $0.11/GB. Sentry Free 5k errors/mo Team $26/mo 50k errors. Datadog Error Tracking $15/host/mo extra.
Data Replication Frequency & SLA Control
Runs in multiple regions for speed and resilience. Enterprise-grade reliability and the hardest to retrofit.Pricing & free-tier limitsAirbyte OSS cron free unlimited. Hevo scheduling included $200/mo plan up to 15-min sync 1-min addon $100/mo. Fivetran hourly free, 15-min $0.02/credit extra per MAR, 1-min custom $0.05/credit. SLA monitors via Monte Carlo $0.30/SLA check/min.
Identity & Access Management (RBAC) for Pipelines
How users sign up, log in, and are authorized. Getting roles and access control right early prevents painful rewrites.Pricing & free-tier limitsKeycloak OSS free unlimited users. OPA free. Auth0 Free 7,500 MAU, B2B Pro $150/mo includes RBAC. Okta Starter $2/user/mo min $500/mo Enterprise $8/user SSO + $2/RBAC. AWS IAM free, Identity Center $0.
Network / VPC Peering for Sources
A piece of your stack you may or may not need, depending on scope. Pick the option that fits — or skip it if your project doesn’t require this capability yet.Pricing & free-tier limitsWireGuard free. Tailscale Free 100 devices Personal Pro $6/user/mo. AWS VPC Peering $0.01/GB processed same region. Transit Gateway $0.05/hr per attachment + $0.02/GB processed. PrivateLink $0.01/hr per AZ + $0.01/GB. Cloudflare Tunnel free unlimited.
Data Masking / PII Tokenization in Transit
A piece of your stack you may or may not need, depending on scope. Pick the option that fits — or skip it if your project doesn’t require this capability yet.Pricing & free-tier limitsVault Transform free OSS. Faker OSS free. Skyflow Free dev 1k tokens Starter $500/mo 100k tokens $0.10/1k after. Privacera $30k/yr starter. Protegrity $2k/mo min. TokenEx Free 10k tokens/mo $0.08/1k after.
Encryption at Rest & In-Transit for Staging
An isolated place to test changes before real users see them. Catches bugs without embarrassing your production users.Pricing & free-tier limitsSOPS + Age OSS free. Let's Encrypt free certs. AWS KMS Free 20k req/mo after $0.03/10k req + $1/key/mo + $0.03/10k crypt. GCP KMS $1/key/mo $0.03/10k ops. Vault Transit free OSS Enterprise HSM $1.50/hr namespace. ACM free, Private CA $400/mo CA.
IP Allowlisting / PrivateLink for Databases
A piece of your stack you may or may not need, depending on scope. Pick the option that fits — or skip it if your project doesn’t require this capability yet.Pricing & free-tier limitsIP allowlist manual free. Confluent PrivateLink add-on $250/cluster/mo. Fivetran PrivateLink $350/mo per connector. StrongDM Free 5 users Team $30/user/mo. Teleport Community free self-host Enterprise Cloud $15/user/mo.
High Availability / Multi-AZ Deployment
Automates testing and deploying your code. Saves enormous time and prevents “works on my machine” releases.Pricing & free-tier limitsK8s HA free OSS. MSK Multi-AZ 3 brokers m5.large $0.21/hr each ~$504/mo per broker = $1,512/mo trio. Astro HA add-on $500/mo 99.99% SLA. Confluent Multi-zone +30% on CKU cost. RDS Multi-AZ db.m5.large $0.342/hr vs single $0.171/hr.
Load Balancing for Ingestion Endpoints
A piece of your stack you may or may not need, depending on scope. Pick the option that fits — or skip it if your project doesn’t require this capability yet.Pricing & free-tier limitsHAProxy OSS free. NLB new AWS Free 750hr/mo 12mo after $0.0225/hr + LCU $0.006. ALB $0.0225/hr + LCU $0.008. Cloudflare LB Free 1 origin Pro $5/mo + $5/LB extra origin $0.25 per 100k req. F5 $2.5k/mo. Global Accelerator $0.025/hr + $0.015/GB.
Caching Layer for API Source Responses
Stores frequent results in fast memory so you serve less from the database. Big lever for speed and cost.Pricing & free-tier limitsRedis OSS free. Upstash Free 10k cmds/day Pay-as-you $0.20/100k cmds storage $0.25/GB. Cloudflare KV Free 1GB + 100k reads/day, Workers Paid $5/mo 10M reads. ElastiCache Serverless Free 100hr t4g.micro trial $0.044/GB-hr + $0.125/1M ECPUs. Redis Cloud Free 30MB Essentials $5/mo 100MB.
Secrets Rotation & Automated Credential Refresh
A piece of your stack you may or may not need, depending on scope. Pick the option that fits — or skip it if your project doesn’t require this capability yet.Pricing & free-tier limitsESO free OSS. AWS Secrets Manager rotation free feature $0.40/secret/mo. Doppler auto-rotation Starter $18/user/mo includes 10 rotations. Infisical rotation free tier 5 rotations Pro $5/user/mo. CyberArk $50k/yr min. Vault dynamic creds $0.50/hr namespace HCP.
Disaster Recovery & Cross-Region Sync
Copies of your data so a bug, hack, or outage doesn’t become permanent loss. Cheap insurance every serious project needs.Pricing & free-tier limitsMirrorMaker2 free OSS. S3 CRR $0.023/GB storage ×2 + $0.02/GB replication fee. Confluent Cluster Linking $0.02/GB transferred. AWS DMS Free t3.micro 750hr trial $0.30/hr after. Veeam cross-region $0.50/workload/day. Databricks DR $0.15/DBU addon.
Compliance & Certification (SOC2/HIPAA/GDPR)
Keeps you secure and compliant with regulations. Required for enterprise customers and handling sensitive data.Pricing & free-tier limitsOpen checklist free. Vanta Free trial 30d Starter $3k/yr 25 employees + SOC2 Type II readiness $25k/yr. Secureframe $7k/yr starter 20 controls. Drata Starter $6k/yr. Coalfire audit $15k-$45k one-time SOC2. HIPAA addon $5k/yr, GDPR $2k/yr, auditor $20k-$50k/yr.
SLA Management & Incident Alerting
Tells you when things break and why. You cannot fix what you cannot see — this is how you keep downtime short.Pricing & free-tier limitsGrafana Alerting OSS free. OnCall Free 1 team. Better Stack Free 10 monitors Starter $24/mo 50 monitors. PagerDuty Free 5 users 1k alerts Pro $41/user/mo includes SLA. Monte Carlo SLA monitoring $500/mo per 10 SLAs $0.10/breach alert. SLO burn $0.05/SLO-hour.
Custom Connector SDK / Connector Builder
A piece of your stack you may or may not need, depending on scope. Pick the option that fits — or skip it if your project doesn’t require this capability yet.Pricing & free-tier limitsAirbyte CDK free OSS unlimited custom. Singer SDK free. Estuary Flow Builder free 1 custom connector Team $200/mo 10 custom. Fivetran SDK free for partners else $500/mo min Private Preview. Airbyte Enterprise Builder $1k/mo per custom connector support $200/hr.
Workflow / DAG Version Control & CI/CD
Where your source code lives and is collaborated on. Non-negotiable for any team or serious project.Pricing & free-tier limitsGitHub Free 2k Actions mins/mo $0.008/min after storage 500MB free. GitHub Team $4/user/mo 3k mins. CircleCI Free 6k credits/wk ~1k mins/wk Team $15/mo 30k credits. Harness Free 2k CI mins/mo Pro $200/mo 5k mins overage $0.01/min. dbt Cloud CI 1k free runs/mo $0.10 extra.
How this enterprise integration and etl elt checklist works.
Each requirement below is something a enterprise integration and etl elt build typically needs. Pick one of the four researched options — recommended, free, cheaper or paid — add your own with "Other", or skip the requirement if your project doesn't need it. Nothing is mandatory; the plan on the right tracks what you've decided so nothing gets forgotten.
Your picks are saved in this browser automatically, so you can come back anytime. Options are researched per build level and refreshed as vendors change their plans — always verify details on the provider's page before committing.