What is a Lakehouse?
Why Modern Data Engineering Moved Beyond Data Lakes and Warehouses
This article is part of the Databricks from Scratch series.
Start from the beginning: Stop Optimising Your Prompts. Fix Your Data Pipelines.
Picture this.
It's IPL ticket booking day. 10 AM. 1 crore fans hit the website at the same moment. Every click, search, failed payment, or successful booking — that's millions of rows of data being generated every second.
Now here's the real question nobody asks: Where does all of that data actually live?
Not just the final booking confirmation, but also all users who retried seventeen times. If we get that storage architecture wrong, you either can't analyse it fast enough, or we can't store enough of it cheaply, or both.
That's the problem the Lakehouse was built to solve
The Two Solutions That Came Before
Before the Lakehouse, engineering teams had two options — and both of them were incomplete.
The Data Warehouse was fast and structured. We could run SQL queries on it, build dashboards, and get answers in seconds. But it only worked with clean, structured data and storage was expensive. We couldn't just dump everything in and figure it out later.
The Data Lake solved the storage problem. Cheap object storage (S3, ADLS, GCS) meant you could throw everything in — raw, unstructured, semi-structured, all of it. But querying it was painful. No schema enforcement or ACID transactions. If two jobs wrote to the same table simultaneously, your data could silently be corrupted. Running analytics directly on a data lake was slow and unreliable.
So most companies ran both, and the data was always slightly out of sync between the two.
Double the infrastructure. Double the cost.
The Lakehouse: One Architecture, Both Strengths
It combines the cheap, scalable storage of a data lake with the performance and reliability guarantees of a data warehouse — in a single system.
It sits directly on open file formats (Parquet, ORC) in cloud object storage. No proprietary lock-in. No copying data between systems. The same data that gets ingested raw can be queried with SQL, processed with Spark, and served to a BI dashboard — all from one place.
Three things make this possible:
An open storage format: Data lives in Parquet files on S3 or ADLS. Any tool that can read Parquet can access it. No vendor lock-in.
A metadata and transaction layer. This is where Delta Lake comes in (more on this in a later article). It adds ACID transactions, schema enforcement, and versioning on top of plain Parquet files. This is what makes the storage behave like a warehouse — reliable, consistent, queryable.
A unified compute engine. Apache Spark runs directly against this storage layer. Whether you're doing batch processing, streaming, machine learning, or SQL analytics, one engine handles all of it.
What This Looks Like in Practice
Going back to the IPL scenario.
With a traditional setup, the raw booking data would land in S3 (data lake), get cleaned by a batch pipeline, then load into Redshift or Snowflake (data warehouse) for the analytics team to query. Two systems. A pipeline that runs every few hours. Analysts working with data that's already 4 hours stale.
With a Lakehouse:
Raw booking events stream directly into Delta Lake tables on S3
A lightweight transformation pipeline cleans and aggregates the data
Analysts query the same tables — live — with SQL
The data science team trains seat-demand prediction models on the same raw data
Everything is one system, one source of truth
No duplication. No pipeline lag. No stale dashboards on match day.
The Medallion Architecture
Most Lakehouse implementations organise data into three layers:
Bronze — raw ingestion. Data lands here exactly as it arrived. No transformations. This is your audit trail, your source of truth for reprocessing if something goes wrong downstream.
Silver — cleaned and conformed. Duplicates removed, nulls handled, schemas validated. This is where most transformation logic lives.
Gold — aggregated and business-ready. The tables your dashboards, reports, and ML models read from. Optimised for query performance.
For our IPL booking system:
Bronze: every raw click event, exactly as it came off the event stream
Silver: deduplicated bookings with validated seat and user data
Gold: aggregated tables — seats sold per match, revenue by category, peak load windows
Why Databricks
Databricks built the Lakehouse architecture and is the primary platform that implements it. Delta Lake — the transaction layer that makes this work — was created by Databricks and is now open source.
When you work on Databricks, you're working with this architecture natively. Your notebooks, your pipelines, your SQL queries — all of it runs against Delta Lake tables organised in this Bronze/Silver/Gold pattern.
Understanding the Lakehouse isn't an optional context. It's the foundation everything else in this series builds on.
What's Next?
In the next article, we'll dive into Apache Spark — the distributed compute engine that powers the Lakehouse architecture.
If you're following along, connect with me on LinkedIn.
And if you're interested in cloud-native technologies, be sure to check out my Cloud Native from Scratch series.
