Building a Modern Lakehouse on AWS: Architecture Decisions That Matter

The Architecture Decision Nobody Talks About

When organizations move to a modern data lakehouse on AWS, they often jump straight into tool selection. Should we use Glue? Or EMR? What about Athena?

The real question comes before that: What is your data processing pattern?

Understanding Your Workload

Before choosing your compute layer, answer three questions:

1. How often does your data change? Streaming vs. batch fundamentally changes your architecture.
2. Who queries the data? Data engineers running transformations vs. analysts running ad-hoc queries need different tools.
3. What is your latency requirement? Near-real-time reporting vs. overnight batch jobs are entirely different problems.

The Three Core Patterns

Pattern 1: Batch-first Lakehouse (Most Common)

If your data arrives in daily or hourly batches, this is your starting point:

• S3 as the storage layer (Bronze → Silver → Gold)
• AWS Glue for ETL and data catalog
• Amazon Athena for ad-hoc SQL queries
• Amazon Redshift Serverless for complex analytics workloads

This pattern covers 80% of enterprise use cases. It's cost-effective, fully managed, and scales without infrastructure management.

Pattern 2: Streaming Lakehouse

When you need near-real-time data (under 5 minutes latency):

• Amazon Kinesis Data Streams or MSK (Managed Kafka) for ingestion
• AWS Glue Streaming or EMR with Spark Structured Streaming
• Apache Iceberg on S3 for ACID transactions on the lake
• Redshift Streaming Ingestion for the warehouse layer

Pattern 3: High-Compute Analytics

For complex ML workloads or large-scale transformations:

• Amazon EMR with Spark for heavy computation
• AWS Glue for simpler transformations
• SageMaker for ML integration

My Recommendation

Start with Pattern 1. Move to Pattern 2 only when you have a proven business need for sub-hour latency. Pattern 3 is for teams with dedicated data engineers who know Spark.

The Glue vs. EMR Decision

This is the most common debate. Here's the simple version:

| | AWS Glue | Amazon EMR | |---|---|---| | Setup | Serverless, zero config | Cluster management required | | Cost | Pay per DPU-hour | Pay per EC2 instance-hour | | Flexibility | Limited (PySpark subset) | Full Spark + ecosystem | | Best for | Standard ETL pipelines | Complex transformations, ML |

Rule of thumb: Start with Glue. Move to EMR when Glue can't do what you need.

Closing Thought

Architecture decisions should be driven by your actual workload, team capabilities, and cost constraints — not by what's newest or most impressive. The best architecture is the simplest one that solves your problem reliably.