# Databricks Architecture: Components, Layers & Guide

> Databricks deployment choices depend on the boundaries between coordination, compute, and persistent storage. Understanding those boundaries helps explain where a workload runs, which systems authorize access, and what remains when its compute stops. It also makes deployment decisions more concrete: selecting serverless compute, choosing a table's storage location, and organizing bronze, silver, and gold datasets solve different problems.

- Canonical URL: https://www.puppygraph.com/learn/databricks-architecture
- Published: 2026-11-13
- Author: Hao Wu

Databricks deployment choices depend on the boundaries between coordination, compute, and persistent storage. Understanding those boundaries helps explain where a workload runs, which systems authorize access, and what remains when its compute stops. It also makes deployment decisions more concrete: selecting serverless compute, choosing a table's storage location, and organizing bronze, silver, and gold datasets solve different problems.

This guide follows those boundaries from a workspace request to a lakehouse table, then examines Delta Lake, Unity Catalog, SQL warehouses, and serverless execution. Deployment examples use AWS terminology; identity and networking details vary by cloud.

## What is Databricks architecture?

Databricks architecture is the arrangement of managed platform services, execution resources, and governed storage that supports data engineering, analytics, and machine learning. The platform connects these responsibilities through workspaces, compute interfaces, and a shared catalog.

A Databricks account organizes the deployment. It manages identities and workspaces, while each workspace provides the collaborative environment for notebooks, jobs, queries, and other assets. A Unity Catalog metastore can serve multiple workspaces within the same region, allowing teams to share governed data without sharing every workspace asset. These relationships are described in Databricks' [high-level architecture](https://docs.databricks.com/aws/en/getting-started/high-level-architecture).

**The control plane coordinates the platform.** Databricks operates the web application and backend management services. A query or job submitted through the workspace uses these services to reach the appropriate execution resources.

**The compute plane processes data.** For classic compute on AWS, resources run in the customer's AWS account and workspace network. Serverless compute runs in a [separate compute plane operated by Databricks](https://docs.databricks.com/aws/en/getting-started/high-level-architecture). Both arrangements place execution outside the control-plane services that manage the workspace.

Storage is a separate architectural concern. Table data, workspace files, execution results, and caches have different purposes and lifecycles. A useful deployment model identifies where each lives instead of treating every persisted object as part of the same data lake.

## How does Databricks architecture work?

Consider a network operator that collects router telemetry and service-inventory records. Its engineers need validated event tables, analysts need service-quality reports, and operations teams need to connect faults to affected customer circuits. The following flow shows how the components cooperate.

**Ingest the source data.** Telemetry files arrive in cloud object storage. Auto Loader can discover and incrementally process those files through Structured Streaming. Its checkpoint tracks ingestion progress so processing can resume after a failure. Inventory records can enter through a separate ingestion workflow, with their own update cadence. Databricks documents this file-processing mechanism in its [Auto Loader overview](https://docs.databricks.com/aws/en/ingestion/cloud-object-storage/auto-loader/).

**Coordinate the work.** Lakeflow Jobs schedules and orchestrates tasks, including notebooks and pipelines. A workflow can ingest events, validate device identifiers, and refresh a reporting table in dependency order. The job describes when tasks run and how they depend on one another; each task uses the compute appropriate to its workload. [Lakeflow Jobs](https://docs.databricks.com/aws/en/jobs/) provides the scheduling and dependency model.

**Execute the transformation.** In a distributed Spark application, a driver coordinates execution and sends tasks to executors. Those executors process partitions of data in parallel. Filtering telemetry, joining inventory, and aggregating regional metrics happen in the execution layer. Apache Spark's [cluster overview](https://spark.apache.org/docs/latest/cluster-overview.html) explains this driver-and-executor arrangement.

**Commit and consume the result.** A transformation writes a Delta table that downstream workloads can read. Unity Catalog supplies the table's identity and access controls, while Delta Lake supplies its transactional representation. An analyst can then query that table through a SQL warehouse, independently of the compute used to produce it.

This flow also helps locate failures. Missing source files call for an ingestion investigation; incorrect identifiers call for transformation checks; an authorization failure calls for a permissions review. Increasing compute capacity only addresses problems caused by insufficient execution resources. Designing the workflow around these responsibilities makes its recovery procedures easier to define.

## Understanding the Databricks lakehouse architecture

A lakehouse brings warehouse-style table management and analytical execution to data held in a lake. Databricks combines open table formats, distributed processing, and governance so engineering, BI, and machine-learning workloads can consume shared datasets. Its [lakehouse overview](https://docs.databricks.com/aws/en/lakehouse/) describes ingestion, curation, and serving as connected parts of that architecture.

Persistent tables provide the shared foundation. One compute resource can prepare telemetry while another serves reports over the resulting tables. Compute can be stopped or replaced without making its local disks the durable home of those tables. Different consumers can also use different representations: detailed events for investigation and aggregates for dashboards.

Sharing a lakehouse does not eliminate transformations or derived datasets. The network operator still needs to normalize device identifiers, reconcile timestamps, and define service-quality metrics. The architectural benefit is that these products can remain within a common storage and governance arrangement, with compute selected for each workload.

Table ownership introduces another distinction. With a [Unity Catalog managed table](https://docs.databricks.com/aws/en/tables/managed), Databricks manages the data lifecycle and maintenance responsibilities. With an [external table](https://docs.databricks.com/aws/en/tables/external), Unity Catalog governs access through the catalog while the organization manages the underlying files and their lifecycle. Both types can participate in the lakehouse, but they assign different operational responsibilities.

The word managed describes that responsibility; it does not by itself identify whose cloud account holds the files. Storage location and table-management type need separate decisions. Likewise, an open format supplies an interoperability foundation, while a particular external reader still needs compatible table features, permissions, and network access.

## Databricks architecture layers explained

Architectural layers describe responsibilities. For planning purposes, the platform can be read as the following functional map:

| **Layer** | **Responsibility** | **Network-operations example** |
| --- | --- | --- |
| Ingestion and orchestration | Bring in source data and coordinate dependent work | Load telemetry and refresh inventory |
| Storage and table management | Persist data and maintain table state | Delta event and topology tables |
| Processing | Execute transformations and analytical queries | Normalize device IDs and aggregate events |
| Governance | Organize assets and control access | Restrict customer details while exposing operational metrics |
| Consumption | Present data to people and applications | Dashboards, investigations, and model training |

Governance applies across this map. [Unity Catalog](https://docs.databricks.com/aws/en/data-governance/unity-catalog/) organizes assets under a `catalog.schema.object` namespace and provides access control, lineage, and auditing. Registering a table gives it a governed identity that consumers can reference independently of its storage path. The catalog is therefore involved when data is produced and consumed, rather than appearing only at the end of a pipeline.

Bronze, silver, and gold describe a different kind of layering: the quality and purpose of datasets. Databricks presents [medallion architecture](https://docs.databricks.com/aws/en/lakehouse/medallion) as a recommended design pattern, not a deployment requirement.

**Bronze preserves incoming records.** For the network operator, this could mean raw telemetry plus ingestion timestamps and source identifiers. Keeping the source representation makes replay possible when parsing logic changes. Retention should reflect the organization's recovery and audit requirements.

**Silver establishes usable detail.** A silver event table can standardize timestamps, deduplicate repeated events, and validate router identifiers. Separate topology tables can preserve routers, logical tunnels, and customer circuits. Invalid records need an explicit disposition, such as quarantine for investigation, so a successful pipeline does not conceal rejected input.

**Gold serves defined consumers.** Regional fault counts and service-quality summaries can support operational dashboards. Their definitions should state the reporting interval, included event types, and treatment of late data.

These layers need not correspond to separate workspaces or compute clusters. Their boundaries express data contracts. Detailed silver topology may remain the right input for an investigation even when a gold dashboard is available, because aggregation can remove the relationships the investigation needs.

## Delta Lake in Databricks architecture

Delta Lake provides a transactional table layer over files in object storage. It combines Parquet data files with a transaction log that describes table state. Databricks uses Delta Lake as its default table format, although the platform also supports other formats and access paths. The [Delta Lake documentation](https://docs.databricks.com/aws/en/delta/) explains its role in batch and streaming workloads.

The transaction log lets a reader determine which files belong to a committed table version. Reading a Delta table therefore involves its metadata and protocol, rather than simply scanning every Parquet file under a directory. This distinction matters when writes replace files or introduce a new table version while other workloads are reading.

Delta Lake's [concurrency model](https://docs.delta.io/concurrency-control/) gives readers a consistent snapshot while writers make changes. With optimistic concurrency control, a writer prepares its changes and validates them against intervening commits before committing. Conflicting changes can fail instead of silently overwriting another writer's work. An ingestion pipeline and a reporting query can consequently operate on the same table with defined read and write behavior.

Schema enforcement provides a second control. Databricks checks writes against the target Delta schema and attempts safe type conversions where supported. A telemetry field that unexpectedly changes representation may require an explicit schema or parsing decision. [Schema enforcement](https://docs.databricks.com/aws/en/tables/schema-enforcement) protects the table's structural contract. Databricks [does not enforce foreign key constraints](https://docs.databricks.com/aws/en/tables/constraints), so checking whether a router identifier exists in inventory still belongs in validation logic.

Table versions also support historical investigation. Engineers can inspect an earlier table state when diagnosing an incorrect transformation or a changed result. However, [time travel depends on retained data and log history](https://docs.databricks.com/aws/en/tables/history). Removing files needed by an older version makes that version unavailable for querying, so time travel requires a retention policy and does not replace a backup strategy.

Delta Lake establishes reliable table state. The pipeline remains responsible for whether that state expresses the right facts, including duplicate handling, event-time semantics, and the meaning of a corrected inventory record.

## Databricks SQL architecture

Databricks SQL serves analytical SQL workloads over lakehouse data. A SQL warehouse is the compute resource that executes those queries. Analysts can use the SQL editor and dashboards, while external BI tools connect through supported interfaces such as JDBC and ODBC. The warehouse consumes tables that can also be used by engineering workloads; it does not require a separate copy simply because the consumer uses SQL. Databricks describes these interfaces in its [SQL documentation](https://docs.databricks.com/aws/en/sql/) and [warehouse connection guide](https://docs.databricks.com/aws/en/compute/sql-warehouse/).

Query execution combines planning with an execution engine. Photon is Databricks' native vectorized engine: it processes supported operations in columnar batches and works with Spark SQL and DataFrame interfaces. Its role is execution, while the query optimizer still plans the work. The [Photon overview](https://docs.databricks.com/aws/en/compute/photon) explains that division without requiring applications to adopt a different query language.

Databricks offers [serverless, pro, and classic SQL warehouse types](https://docs.databricks.com/aws/en/compute/sql-warehouse/warehouse-types). Their infrastructure placement and management capabilities differ. Selecting a warehouse type should follow the workload's operating requirements, while choosing its capacity requires understanding both individual queries and concurrent demand.

Those are different sizing problems. A large query that spills intermediate data may need more resources within a cluster. A morning burst of dashboard requests may need additional capacity for concurrent queries. Databricks' [warehouse sizing and scaling guidance](https://docs.databricks.com/aws/en/compute/sql-warehouse/warehouse-behavior) distinguishes cluster size from the number of clusters and recommends using query profiles and queue metrics to identify the constraint.

Caching can reduce repeated storage reads. The [disk cache](https://docs.databricks.com/aws/en/optimizations/disk-cache) keeps local copies of fetched Parquet data on supported compute, including data used by Delta tables. These copies accelerate access while the persistent tables remain in storage. In the network example, telemetry ingestion and dashboard serving can therefore have separate compute lifecycles while sharing the same governed data products.

## Databricks serverless architecture

Serverless compute shifts infrastructure provisioning and operation to Databricks. The platform supplies and scales execution resources, while users submit workloads and configure their data access. Databricks provides serverless options for notebooks, jobs, pipelines, and SQL warehouses, with the available configuration depending on the workload. The [compute overview](https://docs.databricks.com/aws/en/compute/) separates these options from customizable classic compute.

The deployment boundary matters. Serverless resources run in the Databricks-managed compute plane. Access to customer resources still requires appropriate connectivity and permissions. Databricks documents the [serverless network connections](https://docs.databricks.com/aws/en/security/network/serverless-network-security/), including the connection between compute and customer storage. Moving execution to serverless should therefore prompt a review of source reachability, regional placement, and outbound access requirements.

Workload compatibility also needs attention. For example, serverless notebooks and jobs expose a supported Spark Connect API surface, with restrictions on low-level APIs and compute customization. A workload that depends on its own initialization scripts or Spark extensions needs to be checked against those restrictions before migration. The [serverless limitations](https://docs.databricks.com/aws/en/compute/serverless/limitations) are specific to the relevant product surface; a restriction on notebook streaming does not automatically describe managed pipelines.

Serverless workspaces add another choice. A classic workspace can use serverless compute, while a serverless workspace comes with serverless compute and Databricks-managed default storage. It can also connect to customer cloud storage. Databricks' [workspace deployment options](https://docs.databricks.com/aws/en/admin/workspace) distinguish the workspace choice from the compute choice.

That storage choice affects interoperability. Tables in default storage can be queried through supported Databricks SQL interfaces, but external Delta and Iceberg clients cannot directly read their underlying files through credential vending. The [default-storage limitations](https://docs.databricks.com/aws/en/storage/default-storage#limitations) make this boundary explicit. An architecture that needs additional engines to read tables directly should settle the storage and access path alongside its compute decision.

## Databricks architecture diagram

A useful Databricks architecture diagram shows ownership boundaries and the paths between components. For an AWS example with customer-controlled table storage, place the control plane and serverless compute in the Databricks-managed environment, classic compute in the customer's AWS account, and the lakehouse tables in S3. Label Unity Catalog's governance role across the consumers.

![Databricks architecture diagram](https://o6xswp5m6ilbks2z.public.blob.vercel-storage.com/blog-learn-d3681f82d392ce4eec525e32.png)

Figure: Compute placement and table storage are separate decisions: both compute planes can access customer-controlled S3, while Databricks-managed default storage changes external-access options.

Read the diagram by following a workload. A scheduled transformation reaches compute, reads its inputs, and commits a table version. A dashboard query reaches a SQL warehouse and reads the resulting table under its access policy. Dashed arrows show coordination, while solid arrows show data access; the control plane is not a mandatory processing stop between every source and its execution engine.

The same diagram can accommodate another engine where a workload calls for it. For the network operator, regional aggregates answer how many faults occurred. An investigation may instead ask which customer circuits depend on a failed router through nested logical tunnels, and which paths establish that dependency. The detailed topology tables preserve the entities and relationships needed to ask that question.

[PuppyGraph](https://www.puppygraph.com/) is the first and only zero-ETL graph query engine that queries existing tables as a graph without loading them into a separate graph database. It is a distributed graph query engine that reads tables in place in SQL databases, data warehouses, and data lakes and lakehouses, including direct reads of open table formats like Iceberg and Delta Lake. Teams connect their sources, define a graph schema mapping tables to nodes and edges, and query with openCypher and Gremlin, while the same tables remain queryable with SQL.

![Comparison of the best data lakehouse solutions](https://o6xswp5m6ilbks2z.public.blob.vercel-storage.com/blog-learn-4b2587a755bbd5c9068284ee.png)

Figure: PuppyGraph queries the existing tables in place, so graph and SQL queries share one copy of the data and no pipeline keeps a graph copy in sync.

Here, the graph schema gives routers, tunnels, and circuits a shared semantic model over the topology tables. The documented [Databricks connection](https://docs.puppygraph.com/getting-started/querying-databricks-delta-lake-data-as-a-graph/) reads eligible Delta tables through Unity Catalog credential vending. This requires configured external access and appropriate privileges; [tables with row filters or column masks are excluded](https://docs.databricks.com/aws/en/external-access/credential-vending#limitations). [Databricks-managed default storage](https://docs.databricks.com/aws/en/storage/default-storage#limitations) does not support this credential-vending path, so storage placement determines whether the additional engine can read those tables directly.

![Comparison of the best data lakehouse solutions](https://o6xswp5m6ilbks2z.public.blob.vercel-storage.com/blog-learn-5820007cae8f617f5bb57a14.png)

Figure: Companies that use PuppyGraph.

![Comparison of the best data lakehouse solutions](https://o6xswp5m6ilbks2z.public.blob.vercel-storage.com/blog-learn-4492653ad4ba877e25923f1c.png)

Figure: Engineering teams at PuppyGraph customers on what they use it for.

## Conclusion

Databricks architecture connects coordinated execution with durable, governed tables. Control and compute planes explain deployment boundaries; Delta Lake explains table state; Unity Catalog explains asset organization and access; and medallion layers explain how datasets become useful to consumers. SQL warehouses and serverless compute then provide workload-specific execution choices. A sound design makes those choices together, including the storage and access requirements of every engine expected to consume the data.

Try the forever-free [PuppyGraph Developer Edition](https://www.puppygraph.com/download-confirmation) and [book a demo](https://www.puppygraph.com/book-demo) with the team to see how openCypher and Gremlin queries run over warehouse and lakehouse tables, with no graph-specific ETL, for relationship analysis alongside existing Databricks workloads.