Apache Iceberg REST Catalog: Architecture & Setup Guide

Sharing Iceberg tables across query engines requires a common way to discover tables and commit changes. The REST catalog protocol standardizes that interface.

This guide explains the architecture, follows a table through reads and writes, and covers the main API endpoints. It then walks through Spark configuration, including authentication, storage access, and checks that establish whether the complete connection works.

What is Apache Iceberg REST catalog?

The Apache Iceberg REST catalog is a standardized HTTP interface for managing Iceberg tables and namespaces. A REST catalog service implements that interface; an engine's REST catalog client calls it. Apache defines the contract in the Iceberg REST Catalog specification, while different projects and vendors provide servers.

An Iceberg catalog resolves a table identifier to its current metadata and coordinates the publication of table changes. REST specifies how clients request those operations. The server's persistence layer, administrative interface, and access-control model remain implementation choices.

For example, Spark might address a table as lakehouse.sales.orders. Here, lakehouse is a locally configured catalog name, sales is the namespace, and orders is the table. Another engine can use its own catalog alias while reaching the same service and table. Iceberg's Spark configuration documentation describes this naming model.

The REST API carries catalog requests and metadata responses. Query execution and bulk file access remain separate responsibilities, which is why configuring a working catalog connection is only part of connecting an engine to a lakehouse.

How does Iceberg REST catalog work?

A typical interaction starts with configuration, loads a table, and then follows the read or write path.

1. Initialize the connection. The client calls GET /v1/config, optionally passing a warehouse identifier. The server returns configuration defaults and overrides. Client settings take precedence over defaults; server overrides take precedence over client settings.
2. Load the table. The client requests the table by namespace and name. The response includes table metadata and its location, with optional table-specific configuration and storage credentials.
3. Read the files. In the usual client-planned path, the engine selects a snapshot, reads its manifest list and manifests, and identifies the files needed for the query. Its storage integration reads those files directly.
4. Commit changes. A writer prepares new data files and snapshot metadata, then sends requirements and metadata updates to the catalog. The server validates the requirements, applies the updates, writes the final table metadata, and publishes the new state atomically.

Iceberg's REST protocol documentation explains the configuration and commit exchange. The important write-path distinction is that a REST client submits metadata changes for the server to apply. It does not upload a finished table metadata JSON file and simply request a pointer swap.

Concurrent writers use optimistic concurrency. If another writer changes the table, a pending operation may need to refresh its starting state and retry after validation. Compatible appends can often reuse prepared files; conflicting operations may fail. Uploading a data file alone never makes it part of the committed table.

Apache Iceberg REST catalog architecture

The architecture has three main components: engine clients, the catalog service, and file storage. Each has a distinct operational role.

Engine clients translate engine-level operations into catalog requests and use Iceberg metadata to interpret tables. The engine still needs a compatible Iceberg integration and a storage implementation, such as S3FileIO for S3. Iceberg's multi-engine support guide explains why runtime libraries must match the engine version.

The catalog service handles table lookup and commit coordination. Its backend persists catalog state, while its authorization layer determines which operations a caller may perform. The HTTP contract does not prescribe one database or deployment topology for every server.

File storage holds table metadata, manifest lists, manifests, and data or delete files. Table metadata describes schemas, partition specifications, and snapshots; manifests identify files belonging to a snapshot. These relationships are defined by the Iceberg table specification.

Several services implement this interface, with different operational choices:

REST is therefore an interface choice, not an alternative to every named catalog service. Selecting a server also means selecting its operating model, recovery procedures, and permission system.

Key features of Iceberg REST catalog

Common table operations. The API provides a shared vocabulary for creating namespaces, listing tables, loading metadata, and committing changes. Engine integrations can reuse that contract across services. The OpenAPI definition describes request bodies, responses, and errors.

Server-provided configuration. A service can supply settings during initialization and return table-specific settings when a table is loaded. This lets the server communicate storage configuration without requiring every job to reproduce it manually. A REST warehouse value may be a logical identifier rather than a bucket path; use the meaning documented by the chosen service.

Storage access delegation. Supporting implementations can vend temporary storage credentials to authorized clients. The client then accesses the table's files using those credentials. The protocol also describes remote signing, where a service signs storage requests without giving the client the underlying credentials. These are delegation mechanisms, and availability depends on the server and client.

Capability discovery. The configuration response can advertise supported endpoints. Features beyond basic table operations, such as views or server-side scan planning, require compatible implementations. A server supporting REST does not automatically support every optional API.

Keep these catalog capabilities distinct from table-format capabilities. Schema evolution, partition evolution, and snapshot history come from Iceberg's metadata model and engine support. REST provides a common route for loading and updating that model; it does not make every engine's feature coverage identical.

Iceberg REST catalog API endpoints

The following routes cover common catalog interactions. Paths are relative to the configured service URI. {prefix} is an optional routing prefix supplied through catalog configuration; omit that path segment when no prefix is configured. Namespace and table identifiers must be encoded according to the REST API specification.

Creating a table and committing an existing table use different routes, even though both use POST. Neither endpoint accepts rows as a substitute for writing data files through an engine.

Error handling also depends on the operation. A failed commit requirement returns 409 Conflict; an authentication or permission failure calls for checking credentials or grants. A timeout can leave the commit outcome uncertain. Let the Iceberg client handle commit-state checks and supported retries rather than blindly resubmitting a write from application code. If the client still reports an unknown commit state, establish the outcome before retrying.

How to configure Iceberg REST catalog

Configuration must establish access to both the catalog and storage. The example below targets Apache Spark 3.5 built with Scala 2.12, using Iceberg 1.12.0. It assumes an existing OAuth2 REST catalog configured to vend S3 credentials. Other authentication methods and storage systems need their corresponding settings.

Prepare the service and identity. Provision a catalog or warehouse using the implementation's administration tools. Configure its storage location and grant a test identity permission to create a namespace, create a table, and read and write that table. The server also needs the storage permissions required to manage metadata and delegate access. Apache Polaris's setup documentation is one implementation-specific starting point.

Supply these environment variables to the Spark driver through your deployment's secret and configuration management:

Configure Spark. Save the following as rest_catalog_check.py. The catalog options follow Iceberg's REST client configuration. The explicit token endpoint keeps authentication routing separate from the catalog service address.

import os
from pyspark.sql import SparkSession

catalog_options = {
    "type": "rest",
    "uri": os.environ["CATALOG_URI"],
    "warehouse": os.environ["CATALOG_WAREHOUSE"],
    "rest.auth.type": "oauth2",
    "oauth2-server-uri": os.environ["OAUTH_TOKEN_URI"],
    "credential": (
        f'{os.environ["OAUTH_CLIENT_ID"]}:'
        f'{os.environ["OAUTH_CLIENT_SECRET"]}'
    ),
    "scope": os.environ["OAUTH_SCOPE"],
    "io-impl": "org.apache.iceberg.aws.s3.S3FileIO",
    "client.region": os.environ["AWS_REGION"],
    "header.X-Iceberg-Access-Delegation": "vended-credentials",
}

builder = (
    SparkSession.builder.appName("rest-catalog-check")
    .config("spark.sql.catalog.lakehouse",
            "org.apache.iceberg.spark.SparkCatalog")
    .config("spark.redaction.regex",
            "(?i)secret|password|token|credential|access[.]?key")
)
for key, value in catalog_options.items():
    builder = builder.config(f"spark.sql.catalog.lakehouse.{key}", value)

spark = builder.getOrCreate()
spark.sql("CREATE NAMESPACE IF NOT EXISTS lakehouse.rest_demo")
spark.sql("""
    CREATE TABLE IF NOT EXISTS lakehouse.rest_demo.connection_check (
        id BIGINT, message STRING
    ) USING iceberg
""")
spark.sql("""
    INSERT INTO lakehouse.rest_demo.connection_check
    VALUES (1, 'catalog and storage connected')
""")
spark.sql("SELECT * FROM lakehouse.rest_demo.connection_check").show(truncate=False)
spark.stop()

Run it with matching Iceberg runtime and AWS bundle versions, following the dependency pairing in Iceberg's AWS integration guide:

spark-submit \
  --master 'local[2]' \
  --packages org.apache.iceberg:iceberg-spark-runtime-3.5_2.12:1.12.0,org.apache.iceberg:iceberg-aws-bundle:1.12.0 \
  rest_catalog_check.py

The access-delegation header requests vended credentials; the server must actually support and supply them. For a catalog using workload identity directly, configure the engine's storage identity instead. The AWS bundle supplies dependencies for S3 access; it does not grant storage permissions.

Validate the complete path. On a fresh table, the script prints one row with the inserted message. Each rerun appends another row. The DDL operations check namespace and table creation; the insert and select exercise file access and commit publication. Listing tables alone would not establish those capabilities.

If initialization fails, check the base URI, token endpoint, warehouse identifier, and grants. If table discovery succeeds but reads or writes fail, inspect storage credentials, network access, and FileIO dependencies. In a distributed deployment, executors also need network access to the file store.

Harden the deployment. Use HTTPS, restrict grants to the intended tables, and verify that tokens and credentials are redacted from Spark's UI and logs. The example extends Spark's redaction pattern to include credential. Test storage-credential renewal with a long-running job. Catalog permissions cannot restrict a separate identity that already has broader access to the bucket.

For a self-managed catalog, include its persistent backend in availability and backup planning. Monitor failed authentication, commit conflicts, and request latency. Assign table maintenance separately: snapshot expiration and orphan-file cleanup need explicit policies and execution, as described in Iceberg's maintenance guide.

Benefits of Iceberg REST catalog

Less catalog-specific integration work. A shared protocol lets engines connect through a common client interface. This reduces the number of backend-specific integrations a platform team must maintain. Catalog migration still requires testing identities, identifiers, permissions, and supported operations; a common API does not transfer those settings automatically.

Central coordination with independent compute. The service gives writers a common commit path while engines perform their own execution. Teams can choose compute for ingestion, SQL analysis, or other workloads without making the catalog responsible for scanning every row. The operational trade-off is that catalog availability remains important for table discovery and commits.

More consistent access administration. Implementations that connect catalog authorization to scoped storage credentials can reduce the distribution of long-lived bucket credentials. Validate this with actual workload identities, including denied operations. Credential vending alone does not establish identical row filters or column masks across engines.

Shared tables can also support relationship queries. PuppyGraph connects to Iceberg through supported REST catalogs and lets users define a graph schema over existing tables. Its Iceberg connection documentation separates catalog settings from storage settings, following the same architecture described above.

For example, account, transaction, and account-link tables can represent entities and relationships for investigating connected accounts. PuppyGraph runs openCypher and Gremlin queries over that model without graph-specific ETL into a separate database. The tables remain in the lakehouse; users supply the mapping from columns to nodes, edges, and properties. The Iceberg graph-query tutorial demonstrates that mapping over cataloged tables.

Conclusion

Apache Iceberg REST catalog standardizes how engines find tables and coordinate changes. A reliable setup still depends on the chosen service's authentication, storage access, commit behavior, and operational ownership. Validate a complete create, write, and read cycle before treating a successful catalog connection as a working integration.

Try the forever-free PuppyGraph Developer Edition and book a demo with the team to see how openCypher and Gremlin queries explore relationships across warehouse and lakehouse tables, with no graph-specific ETL, using Iceberg tables accessed through a supported REST catalog.