Open Lakehouse: Architecture, Benefits & How It Works

A lakehouse puts warehouse-style tables on cheap object storage, and most of its value rests on one promise: a single governed copy of the data that every workload can use. Whether that promise holds depends on who controls the layers above the storage. If the table format, the catalog, or the only engine allowed to write the tables belongs to one platform, every other tool reaches the data through that platform or not at all.

This guide explains what makes a lakehouse open, how one works and is put together, its core features, how it differs from a platform-bound lakehouse, the role of open table formats and Apache Iceberg in particular, how to build one, and the benefits and limitations to weigh before you do.

What is an open lakehouse?

An open lakehouse is a lakehouse in which every layer between the files and the query is defined by an open specification: data files in an open columnar format, tables described by an open table format, and a catalog reachable through an open protocol, so that compute can come from any engine that implements those standards. No layer requires a particular vendor's engine to read or write the data.

Openness at the file level was part of the architecture from the start. The CIDR 2021 paper that set out the architecture, from Databricks, characterizes a lakehouse by open direct-access formats such as Parquet and ORC, together with first-class support for machine learning and data science. Open files alone leave the hard questions unanswered, though: which files make up the current version of a table, how two writers avoid corrupting it, and where an engine finds it. Those answers live in the table format and the catalog, and a lakehouse whose answers are proprietary is open only at the bottom layer.

The term is used loosely, so a practical test helps. A lakehouse is open to the degree that you can add an engine, or replace one, without migrating data or rewriting table metadata.

How does an open lakehouse work?

An open lakehouse separates three things a database keeps together: where the bytes live, what counts as the table, and which engine runs the query. Every interaction goes through shared metadata rather than through an engine.

The write path. A Spark job appending a day of events writes new Parquet files to object storage, then writes metadata describing them: manifests that list the files with per-file statistics, and a new snapshot that references those manifests. The commit itself is one atomic step in the catalog, which moves the table's pointer from the old metadata to the new. The Iceberg table spec puts it directly: "An atomic swap of one table metadata file for another provides the basis for serializable isolation." If another writer committed first, the swap fails and the writer retries against the newer version.

The read path. A Trino query behind a dashboard asks the catalog for the table's current metadata, reads the snapshot's manifests, skips files whose partition values or column statistics cannot match the filter, and scans only what remains. It always reads a committed snapshot, so it never sees half of a write, and it takes no locks.

Engines share tables through metadata. Because both paths run through the same metadata and the same catalog, the writer and the reader need not be the same engine or come from the same vendor. A Flink job can stream into a table that Spark compacts overnight, Trino serves to analysts, and a data scientist reads from Python with PyIceberg.

The table is defined by its metadata, and that property is what the rest of an open lakehouse builds on.

Open lakehouse architecture

An open lakehouse is a stack of five layers. Openness is decided in the middle three, the file format, table format, and catalog, each of which is defined by an open specification. The storage below them only has to hold files, and any engine above them can read and write the tables once it implements those specifications.

Object storage. Amazon S3, Azure Data Lake Storage, Google Cloud Storage, or an S3-compatible store on premises. It is durable, inexpensive, and scales independently of compute, and it knows nothing about tables.

File format. Parquet is the usual default, with ORC and Avro also supported. Columnar layout lets an engine read only the columns a query touches, and per-column statistics let it skip data before reading it.

Table format. Iceberg, Delta Lake, or Hudi turns a directory of immutable files into a table with a schema, a history of snapshots, and transactional commits.

Catalog. The catalog maps table names to their current metadata and is the point where commits become atomic. Open catalogs expose a standard interface, most often the Iceberg REST protocol, which puts catalog logic behind a common HTTP API so that "a single client implementation works with any compliant server." The catalog is also increasingly where access control happens: under the protocol's credential vending, the catalog hands an engine "short-lived, table-scoped storage credentials", so storage permissions follow catalog permissions.

Compute engines. Spark and Flink for batch and streaming pipelines, Trino, Presto, and Dremio for interactive SQL, DuckDB for local analysis, and cloud warehouses such as Snowflake and BigQuery reading the same tables. Each scales, and is billed, on its own.

The arrangement inverts the warehouse. In a warehouse the engine owns the storage; in an open lakehouse, storage, format, and catalog form a shared substrate and every engine is a client of it.

Core features of an open lakehouse

The first four features come from the table format and its catalog, so any lakehouse has them, open or not; they are described here in Iceberg's terms because the details differ between formats. The fifth is the one openness adds.

ACID transactions on object storage. Atomic catalog commits and snapshot isolation give concurrent writers and readers the guarantees a database provides, on storage that offers none of them natively.

Schema evolution without rewrites. Columns can be added, dropped, renamed, or reordered as metadata-only changes. Iceberg selects columns in data files by field ID rather than by name, so a rename never misreads old files.

Partitioning decoupled from queries. Iceberg's hidden partitioning derives partition values from column transforms, so queries filter on the data itself (an event_time range, not a separate event_date column), and Iceberg allows partitioning to be evolved as data volumes change without rewriting existing files.

Snapshots and time travel. Every write produces a snapshot, and earlier snapshots remain queryable until they are expired. That supports reproducible training sets, audits, and rollback after a bad write; time travel in Iceberg covers the mechanics.

Multi-engine read and write. Any engine that implements the format and the catalog protocol can both query and commit to the same table, which the atomic commits and snapshot isolation described above make safe.

Open lakehouse vs traditional data lakehouse

"Traditional" here means a lakehouse whose data sits in object storage but whose table format, catalog, or write path is controlled by one platform. Many commercial lakehouses started that way and have opened layers since, so the comparison describes two ends of a spectrum rather than two product categories.

The row that matters most is who controls the catalog, because the catalog decides which engines may commit. Many platforms now expose their tables through an Iceberg REST endpoint, and some accept writes from external engines through it. In an open lakehouse, multi-engine writes follow from the standards you adopt; in a platform-bound one, they are whatever the platform's endpoint allows. The platform-bound model gives in return a single operator responsible for performance, maintenance, and security, which is a defensible choice for a team that has standardized on one platform and expects to stay there.

Open table formats in an open lakehouse

Open table formats are the layer that turns a data lake into a lakehouse. Three are widely adopted, and a fourth has grown out of stream processing. Each reflects where it came from.

Apache Iceberg is organized around an engine-neutral specification and a catalog protocol, and has attracted a wide set of independent engine and catalog implementations. The next section covers it in detail.

Delta Lake records each commit in an ordered transaction log alongside the data. It originated at Databricks, is now a Linux Foundation project, and has its deepest integration with Spark, with connectors for Flink, Trino, and other engines.

Apache Hudi describes its table format as "purpose-built for high performance writes on incremental data pipelines." Its copy-on-write and merge-on-read table types and its indexing are built for efficient upserts and deletes, which suits ingestion such as change data capture.

Apache Paimon began in the Flink community as Flink Table Store. It stores primary-key tables in LSM trees: writers flush sorted files and compaction merges them, so a table can absorb a continuous stream of updates and serve its changes to streaming readers. Its closest integrations are with Flink and Spark.

The formats solve the same core problem, transactional tables over immutable files, so interoperability tools can translate between them. Apache XTable (incubating) writes metadata for one format over another format's existing data files without rewriting them, and UniForm generates Iceberg metadata over the data files of Delta tables. Paimon can likewise publish Iceberg metadata over its own data files, so Iceberg readers can query a Paimon table that Paimon still writes. Translation narrows the decision without removing it, because features differ between formats and the translated side is usually limited to reads.

In practice the choice follows the engines that must write to the tables and the shape of the writes. As a rule of thumb, broad engine neutrality points to Iceberg, a Spark-centered platform to Delta Lake, record-level upserts at high frequency to Hudi, and Flink-centered streaming updates to Paimon.

Role of Apache Iceberg in an open lakehouse

Iceberg is the format many open lakehouses standardize on, largely because of the project's posture: it defines a specification first and implementations second, and leaves the choice of catalog and engine to the user.

A versioned table specification. The spec is adopted in versions. Version 1 defines analytic tables on immutable files, version 2 adds delete files for row-level changes, and version 3 adds new data types, column default values, row lineage, binary deletion vectors, and table encryption keys. Support for a version varies by engine, so a v3 feature is usable only when every engine that touches the table implements it.

A catalog protocol. The REST protocol moves commit logic to the server: a client "never writes a metadata file and swaps a pointer itself", and the server validates each commit's requirements against the table's current state. Catalogs that expose it include Apache Polaris, which graduated to an Apache top-level project in February 2026, the open-source Unity Catalog project, and managed services such as Amazon S3 Tables and Google's Lakehouse runtime catalog (formerly BigLake metastore).

Wide engine support. The project maintains Spark and Flink connectors in its main repository, and its guidance for other engines is to build support by importing the Iceberg libraries, as Trino and Presto do, so that each engine's Iceberg support evolves with the engine.

What Iceberg standardizes is the table. Compaction, snapshot expiration, and cleanup of orphaned files remain operational work, done by whichever engine or service the team assigns; Amazon S3 Tables, for example, runs compaction and snapshot management by default.

How to build an open lakehouse

The steps run in dependency order. Workloads decide which engines you need, those engines constrain the format and catalog, and the format and catalog are the decisions that are expensive to reverse. Engines come last because, once the rest is in place, they are the easiest part to add or replace.

Start from workloads. List what the data must serve: batch transformation, streaming ingestion, interactive SQL, machine learning pipelines, and any analysis of how entities connect. The engines those workloads need determine which format and catalog support you require.

Choose the table format. Apply the criteria from the formats section. Most organizations standardize on one format for new tables, even when translation tools let them read others.

Choose the catalog deliberately. The catalog is the hardest layer to change later, because every engine's configuration and every access policy points at it. Prefer one that implements the Iceberg REST protocol, so a new engine connects without custom integration, and decide early whether it will vend storage credentials and enforce access policy.

Assign table maintenance. Schedule compaction of small files, snapshot expiration, and orphaned file cleanup, and name the job or service that owns each. Streaming ingestion makes this urgent, since frequent small commits produce many small files.

Add engines per workload. With tables and catalog in place, each new workload becomes a choice of engine. Graph traversal is one such workload: questions about how accounts, devices, services, or suppliers connect, answered by following relationships across many hops.

PuppyGraph adds that workload as one more engine on the lakehouse. It is a distributed graph query engine that connects to SQL databases, data warehouses, and data lakes and lakehouses, including direct reads of open table formats like Iceberg and Delta Lake through their catalogs, and maps the existing tables to nodes and edges through a graph schema you define. There is no ETL into a separate graph store, so the tables remain the system of record and stay queryable by every other engine.

Queries in openCypher and Gremlin compile into a plan of node and edge operators that runs in parallel across PuppyGraph's compute nodes, so graph compute scales independently of the lakehouse's storage, like any other engine on it. Because the query is represented as graph operators end to end, the engine optimizes specifically for multi-hop traversals and pattern matching. The graph schema also serves as a semantic model of the domain, so analysts and applications ask questions in terms of entities and relationships. For example, AMD builds a graph layer over an Apache Iceberg lake this way, spanning tickets, code, logs, and telemetry.

Benefits of an open lakehouse

Each benefit below follows from one property: a table defined by open metadata belongs to no engine. A team gets them in proportion to how many layers of its stack are open, the spectrum described in the comparison above.

Engine choice without migration. Adopting a faster SQL engine, a streaming processor, or a specialized engine for a new workload means pointing it at the catalog. The data stays where it is.

One copy for many workloads. BI, data science, streaming, and machine learning read the same governed tables, which removes the pipelines and copies that exist only to move data between systems, and the drift between those copies.

Cost that follows usage. Storage is billed at object-storage rates, and each engine's compute scales up, down, or off independently, so a heavy monthly job does not set the size of an always-on cluster.

Governance in one place. When engines reach storage through the catalog, permissions defined there apply across them, rather than being reimplemented in each engine.

Data that outlives its tools. Tables written to a public specification remain readable by future engines without depending on any one vendor.

Challenges and limitations of an open lakehouse

Each limitation below comes from work that spans engines, which no single engine or vendor owns in an open lakehouse: table maintenance, feature support, policy enforcement, catalog operations, and tuning.

Operations become your responsibility. A managed platform hides compaction, file sizing, and snapshot cleanup; an open lakehouse exposes them. Neglected tables accumulate small files and stale snapshots, and query performance degrades gradually.

Engine support is uneven. Engines implement the formats at different depths. Write support, row-level deletes, and newer spec versions arrive engine by engine, and the usable feature set of a shared table is what its least capable engine supports.

Governance covers only what goes through the catalog. Catalog policies bind engines that request access through the catalog and honor its decisions. A job holding direct bucket credentials bypasses them, and how row- and column-level policy reaches a third-party engine depends on the engine and the deployment.

The catalog becomes critical infrastructure. Every read and every commit depends on it, so its availability, scaling, and backups need the attention a production database receives. Organizations that end up with several catalogs also inherit the work of keeping them consistent.

Performance tuning has no single owner. Sort order, file size, and partitioning are set by whoever writes the table, while the engines reading it each have their own optimizer.

These are the costs of choice. A platform-bound lakehouse absorbs them into one vendor's product; an open lakehouse asks a team to own them in exchange for the freedom to change engines.

Conclusion

An open lakehouse extends openness from the files to every layer above them: a table format defined by a public specification, a catalog reachable through a standard protocol, and engines that read and commit to the same tables as peers. Iceberg and its REST catalog protocol are where most of that standardization now happens. The result is a data platform where adding a workload means choosing an engine, and the main cost is taking ownership of the maintenance and governance a single vendor would otherwise handle.

Try the forever-free PuppyGraph Developer Edition and book a demo with the team to see how openCypher and Gremlin queries traverse warehouse and lakehouse tables, with no graph-specific ETL, adding graph analysis to an open lakehouse as one more engine on the tables it already holds.