Apache Iceberg Architecture: Components & How It Works

An analytical table needs a precise answer to a basic question: which files belong to this version of the data? That answer must remain consistent while ingestion jobs append records, maintenance jobs rewrite files, and queries read the table. Apache Iceberg makes table state explicit through metadata, allowing readers and writers to coordinate around committed versions.

Understanding that metadata explains much of Iceberg's behavior, from pruning files before a scan to reading historical snapshots. This guide follows the catalog, metadata, and data layers, then traces writes and queries through them. It also explains where specialized query engines can use the same underlying tables.

What is Apache Iceberg architecture?

Apache Iceberg architecture is the organization of table metadata, file references, and commit operations that turns files in storage into a versioned analytical table. Apache Iceberg is an open table format. Compute engines supply query execution, while storage systems hold the table's files.

A useful distinction is between a file format and a table format. Parquet describes how values are encoded inside a file. Iceberg describes how files participate in a table, including its schema, partitions, and historical states. A directory containing Parquet files does not, by itself, provide those table-level semantics.

This separation lets compatible engines share tables while implementing their own execution strategies. It also places a clear responsibility on each integration: the reader must interpret Iceberg metadata and supported table features correctly. Reading all the Parquet files under a storage prefix is insufficient because that prefix can contain historical files or files from an unsuccessful write.

Apache Iceberg architecture overview

The usual read path starts with a catalog lookup. An engine resolves a table name to its current metadata, chooses a snapshot, and follows that snapshot's references to the files needed for the query.

The main reference chain is:

Catalog → table metadata → snapshot → manifest list → manifests → data and delete files.

Each step narrows the question. The catalog identifies the table's committed metadata. The snapshot identifies a table state. The manifest list and manifests describe its file membership and provide statistics for scan planning. The engine reads the relevant files and evaluates the query.

This is a hierarchy of references, not a pipeline that moves every row through the catalog. In a typical deployment, the engine contacts the catalog for table information and reads files from storage using the configured access credentials.

The three layers of Apache Iceberg architecture

The three-layer model separates responsibilities. It does not require three independently deployed services: metadata files and data files commonly live in the same storage system.

The compute engine operates across these layers. It uses catalog and metadata interfaces to plan work, then reads storage to execute that work. Catalog choice, file layout, and engine behavior therefore affect different parts of the same query.

For example, adding workers may help an expensive aggregation but will not repair a missing metadata object. Likewise, a successful catalog connection does not prove that the engine can access the data files. Keeping these responsibilities separate makes troubleshooting more targeted.

Apache Iceberg catalog layer

An Iceberg catalog provides named access to tables and participates in publishing changes. Iceberg's custom catalog documentation describes two central operations: retrieving a table's metadata location and updating that location atomically.

For a table such as lakehouse.sales.orders, the engine's configured catalog resolves the table identity. The catalog does not need to store every data-file record itself; the metadata hierarchy holds that information. This keeps table discovery separate from the potentially much larger inventory of files.

Catalog implementations use different backends. Hive Metastore and AWS Glue are established options; REST catalogs expose a standardized HTTP interface. The REST Catalog specification defines the interface, rather than prescribing one database or storage service behind it. A REST endpoint and its backend are separate architectural choices.

During a commit, the catalog implementation must prevent a writer from silently replacing changes committed by another writer. The precise mechanism depends on the implementation. The important contract is atomic publication of the new table state.

Catalog access and storage access also need to work together. Some deployments use separately configured storage credentials; some REST catalogs support storage credential delegation. An engine that can list a table still needs authorization to read its referenced objects. Plan both access paths when connecting another engine to the lakehouse.

Apache Iceberg metadata layer

The metadata layer holds the table definition and the information needed to locate its contents. Several artifacts contribute to this, and their names describe different levels of the hierarchy.

Table metadata files. The JSON metadata records schemas, partition specifications, table properties, and snapshots. It identifies the current snapshot and retains references to other available snapshots. A committed metadata update creates a new metadata file; it does not edit the previous file in place.

Snapshots. A snapshot describes a table state and references its manifest list. It is an entry in table metadata, not a second copy of the table's rows. Successive snapshots can share unchanged files. These structures are defined in the Iceberg table specification.

Manifest lists. A manifest list identifies the manifests associated with a snapshot and includes partition summaries. The planner can use those summaries to skip manifests that cannot contribute matching data.

Manifest files. A manifest records data files or delete files, along with information used to interpret and filter them. For data files, this includes partition values and available column statistics. Manifest lists and manifests use Avro encoding. Iceberg's scan-planning documentation explains how the two metadata levels reduce the files inspected during planning.

These distinctions matter when diagnosing a slow query. Time spent reading many manifests is planning work; time spent decoding rows is execution work. A selective predicate enables file pruning only where the layout and available statistics allow files to be excluded.

Metadata also supports table evolution. Iceberg assigns columns stable field IDs, so a rename preserves the column's identity. Supported schema changes can update metadata without rewriting existing data files. A metadata-only change, such as renaming a column, need not create a new data snapshot. Partition evolution similarly allows old and new layouts to coexist: new writes use the changed specification while older files retain their original layout.

Changing a partition specification therefore does not reorganize historical data immediately. If the goal is to improve the layout of existing files, that requires a separate rewrite. The metadata change and the physical maintenance operation solve different problems.

Apache Iceberg data layer

The data layer contains the stored records and, where applicable, information that marks records as deleted. Files are immutable once written. Updates publish a new table state that references new files and may stop referencing older ones.

Two common approaches to row changes are copy-on-write and merge-on-read. Copy-on-write rewrites affected data files with the resulting records. Merge-on-read records changes using new data and deletion information that readers reconcile. Iceberg exposes these choices through its write configuration, with support depending on the engine and operation.

The trade-off is where work occurs. Rewriting files adds write work but can simplify subsequent reads. Keeping separate deletion information can reduce rewriting, while readers must apply that information to return the correct rows. An update-heavy workload may therefore need a different maintenance strategy from an append-only event table.

Position deletes identify rows by data-file path and row position. Equality deletes identify rows by values in specified fields, subject to Iceberg's applicability rules. Readers must apply the deletes relevant to the selected snapshot and data files; simply scanning the underlying data files can return logically deleted records. The row-level delete specification defines these semantics.

Storage can retain files that the current snapshot no longer uses because older snapshots still reference them. Conversely, unsuccessful jobs can leave unreferenced objects. Snapshot expiration and orphan-file cleanup address these different cases. Neither should be replaced with a rule that deletes objects solely because their modification time looks old.

Data files and file formats in Apache Iceberg

An Iceberg table combines several file types. Their roles are more useful to understand than their extensions alone.

Parquet and ORC data files. Both organize values by column, which supports analytical queries that read selected attributes. Parquet provides column-oriented encoding and compression. ORC organizes columnar data into stripes and maintains internal information that readers can use to narrow scans. The engine's file reader determines how these capabilities contribute to execution.

Avro files. Iceberg also supports Avro as a data-file format, while using Avro for manifests and manifest lists. Those are separate roles: an Avro manifest contains file-level records about the table, not its business rows. A filename extension alone therefore cannot tell you whether an object belongs to the metadata hierarchy or contains table data.

Puffin files. Puffin is a container for typed binary payloads and the metadata needed to interpret them. It can hold auxiliary statistics, such as sketches used to estimate distinct values. These are different from the basic file statistics carried in manifests.

Puffin can also hold deletion vectors: bitmaps identifying deleted row positions within a data file. This role matters because deletion information affects query correctness. A reader may be able to ignore an optional statistics payload, but it cannot ignore an applicable deletion vector and still return the correct table contents.

Not every table uses Puffin, and not every engine supports every table-format feature. Before enabling a new write feature, check the capabilities of all readers that need the table. Sharing an open format works when the participating engines agree on the features actually present in its files.

How Apache Iceberg layers work together

Consider an ingestion job appending orders to a table. The writer starts from committed metadata and prepares a new state. The existing table remains readable while this work proceeds.

1. Write new data files. The job writes the incoming records to storage. These objects are not part of the committed table merely because they exist.
2. Prepare file metadata. The writer records the new files in manifests and prepares the snapshot's manifest list, reusing existing metadata where appropriate.
3. Prepare table metadata. New metadata records the proposed snapshot and its relationship to the table's history.
4. Commit the change. The catalog implementation atomically publishes the update if the commit's requirements are satisfied.

Iceberg uses optimistic concurrency. If another writer commits first, the original writer must refresh its view and validate whether its operation can still succeed. Compatible operations may retry; conflicting assumptions can require the operation to fail.

For example, an append may remain valid after another append. A rewrite that assumes particular source files still exist needs different validation if another job has already replaced them. Retry therefore requires validating the operation against the refreshed table state.

A reader continues using its selected snapshot while newer states are committed. This is why writing files and publishing their membership are separate operations. The atomic commit makes a complete state visible, while immutable files let readers finish work against an earlier state.

How query execution works in Apache Iceberg

A query engine turns an Iceberg table reference into scan tasks, then applies its own execution operators. Consider a Spark SQL query over an orders table configured in an Iceberg catalog:

SELECT customer_id, SUM(total_amount) AS daily_total
FROM lakehouse.sales.orders
WHERE ordered_at >= TIMESTAMP '2026-09-01 00:00:00'
  AND ordered_at < TIMESTAMP '2026-09-02 00:00:00'
GROUP BY customer_id;

Assume ordered_at is a timestamp and total_amount is numeric. The engine loads the table and selects the snapshot for this read. It then determines which files might contain matching orders.

If the table is partitioned by day of ordered_at, hidden partitioning lets Iceberg derive partition filters from the timestamp predicate. The query does not need a separate, user-maintained date column. Metadata filtering can exclude irrelevant manifests and data files before the engine schedules their reads.

Available lower and upper column bounds provide another exclusion test. A file whose timestamp range falls entirely outside the requested day cannot contribute rows. Overlapping bounds identify candidate files whose remaining predicates still need evaluation. This distinction explains why useful clustering can improve pruning while randomly distributed values may produce broad bounds.

The engine reads the required columns from candidate files, applies applicable deletions and remaining filters, and executes the aggregation. File-format capabilities can further reduce work within selected files. Iceberg supplies table semantics and scan information; the engine supplies scheduling, joins, aggregation, and resource management.

This creates two useful performance questions: how much work is needed to plan the scan, and how much data survives pruning? Iceberg's maintenance guidance distinguishes rewriting small data files from rewriting manifests. The former improves physical file layout; the latter can improve metadata organization. Neither substitutes for understanding the query's filters and data distribution.

Apache Iceberg enables a decoupled storage-and-compute architecture: tables remain in shared storage, while compatible engines provide the compute for different workloads. Developers can use the same underlying tables for SQL analytics, data processing, and relationship analysis without maintaining a separate dataset for each engine. For multi-hop relationship traversal, this means developers no longer need to export data from Iceberg and load it into a graph database. They can bring a graph query engine to the existing tables, extending the same architectural principle from tabular analytics to graph workloads.

PuppyGraph is the first and only graph query engine for Apache Iceberg. It connects to the existing Iceberg catalog and storage and lets users define a graph schema over the tables. For example, accounts, devices, and payments could become entities and relationships for tracing connections across several hops. The Iceberg graph-query tutorial demonstrates mapping tables to nodes and edges and querying them with openCypher and Gremlin. This adds a semantic model for relationship analysis while the underlying records remain in Iceberg.

PuppyGraph compiles a graph query into a plan of node and edge operators that runs in its own distributed engine. Because the query is represented as graph operators end to end, the engine optimizes specifically for multi-hop traversals. Within the Iceberg architecture, this makes PuppyGraph another compute layer over the stored tables. SQL engines can continue serving tabular workloads while PuppyGraph executes relationship queries over the same underlying data.

The practical reason to add it is a workload that repeatedly follows relationships: accounts sharing devices, payments connecting those accounts, or paths through ownership records. Defining that model once gives analysts reusable entity and relationship names. The direct-query path avoids maintaining a separate graph ingestion pipeline, while Iceberg remains responsible for table state and file organization. Deployment still requires access to both the catalog and storage, as described in the Iceberg connection documentation.

How time travel works in Apache Iceberg

Time travel selects an older snapshot and follows its file references through the same metadata hierarchy. The engine reads the files belonging to that state, including any applicable deletion information. It does not need to reconstruct the table by replaying every intervening change.

Iceberg's Spark query documentation supports snapshot-ID and timestamp-based selection. For a table with retained history spanning the requested time:

SELECT order_id, customer_id, total_amount
FROM lakehouse.sales.orders
TIMESTAMP AS OF '2026-09-01 00:00:00';

Timestamp selection resolves to the historical table state at that time. Selecting a known snapshot ID is useful when recording the exact input to a reproducible analysis. The timestamp concerns table history, not a filter on a business column such as ordered_at: a late-arriving order may have an earlier event time than the commit that first made it visible.

Retention defines the boundary. Once a snapshot is expired, it is no longer available for time travel, and files no retained snapshot needs may be removed. Set retention around reproducibility and audit requirements, along with storage cost. Snapshot history depends on keeping its referenced objects accessible; it does not replace a recovery strategy for storage loss.

Conclusion

Apache Iceberg coordinates table identity, versioned metadata, and stored files so that engines can agree on what a table contains. The catalog publishes changes, the metadata hierarchy describes snapshots and narrows scans, and data files hold the records. Understanding those responsibilities helps explain both correctness and performance: commit coordination, metadata planning, file layout, and engine execution each deserve separate attention.

Try the forever-free PuppyGraph Developer Edition and book a demo with the team to see how openCypher and Gremlin queries explore relationships across warehouse and lakehouse tables, with no graph-specific ETL, using your existing Iceberg catalog and storage.