What Is Medallion Architecture?

Reliable analytics depends on knowing what has happened to the data before a query reads it. Has it simply arrived from a source system, passed validation, or been shaped around an agreed business definition? Medallion architecture makes those stages explicit, giving engineers and consumers a shared way to distinguish incoming records from reusable datasets and finished data products.

This guide explains the Bronze, Silver, and Gold layers, follows a worked example through them, and covers the implementation decisions that determine whether the pattern improves reliability or adds unnecessary maintenance.

What is medallion architecture?

Medallion architecture is a data design pattern that organizes data into three stages of refinement: Bronze for raw ingestion, Silver for validated and standardized data, and Gold for business-ready datasets. The layers describe what processing data has undergone and how consumers can use it. Databricks documents the pattern as progressive improvement in data structure and quality.

A lakehouse supplies the storage, table management, and query capabilities on which these stages run. Medallion architecture supplies the organization of the processing work. The two concepts answer different questions: where analytics operates, and how data becomes ready for it.

The pattern does not require three separate platforms. Layers can be represented through schemas, tables, or separate storage boundaries, depending on the implementation. Microsoft Fabric's guidance, for example, describes deployments using lakehouses for each layer or a combination of lakehouses and a warehouse.

The important boundary is the promise each dataset makes to its consumers. A Gold label is useful only when the team can explain its business meaning, validation rules, and refresh expectations.

How does medallion architecture work?

Each layer reads upstream data and publishes a more usable representation. Ingestion preserves source information in Bronze. Transformation jobs prepare Silver datasets. Business modeling produces Gold outputs for particular reporting or application needs.

Consider an order timestamp delivered as text. Bronze preserves the received value. Silver parses it using a documented time-zone rule and separates records that cannot be interpreted. Gold assigns the order to the reporting period required by the business. These are different responsibilities, even when one processing engine executes them all.

The dependencies form a branching pipeline. One validated order table might support a sales report, a fulfillment analysis, and a customer model. Each consumer can reuse the shared cleaning work while applying its own definitions downstream.

The execution platform supplies the transactional guarantees. For example, Delta Lake uses concurrency control to give readers consistent table snapshots and coordinate concurrent writes. Calling tables Bronze, Silver, and Gold does not make a sequence of jobs one atomic transaction. A Silver refresh can succeed while a downstream Gold refresh fails, so the pipeline must expose freshness and recovery status.

The three layers of medallion architecture

Bronze: preserve what arrived. This layer retains source records with enough context to interpret their arrival, such as the source identifier, ingestion time, file name, or change position. Databricks' layer descriptions emphasize preserving source structure and ingestion metadata so data can be reprocessed without fetching it again.

For an order feed, keep the original order identifier and source version alongside the payload. Avoid converting an unrecognized status into a familiar one merely to make ingestion succeed. Preserving that value lets an engineer investigate a source change later. Raw retention still needs an explicit policy; it does not imply keeping every record indefinitely.

Silver: establish reusable records. Here, the team defines keys, normalizes representations, resolves duplicates, and applies validation rules. A useful Silver order dataset has a declared grain, such as one current row per order. A separate history dataset might retain every accepted order version.

Missing customer identifiers need a deliberate treatment. Depending on the use case, preserve an unresolved reference, route the record for correction, or block publication. Silently assigning unrelated orders to one generic customer would create false relationships, even if the resulting table passed a non-null check.

Gold: publish a business interpretation. Gold tables serve defined consumers. They can contain aggregates, dimensional models, or detailed records prepared for a specific application. An order-level fact table can be Gold when its business definitions and intended use are established; aggregation is not the only qualifying transformation.

For a sales dataset, define whether the measure includes cancellations, refunds, taxes, and shipping. These decisions determine what a dashboard means. A technically valid sum can still answer the wrong business question.

Treat these as distinct responsibilities. More specialized data is not automatically more useful: an investigator may need detailed Silver records that a Gold aggregate has intentionally removed.

How data moves through the medallion architecture

Movement between layers means reading an upstream representation and producing a downstream one. The processing can be scheduled batch work, incremental micro-batches, or streaming. Choose the cadence from the consumer's freshness requirement and the source's delivery behavior.

Track progress explicitly. A pipeline needs to know which files, offsets, or source changes it has processed. It also needs a restart strategy. A retry after a partial failure should not duplicate accepted orders or add the same sales amount twice. Design writes to be idempotent: replaying the same input produces the same intended result.

Apply changes in source order. Change data capture can deliver inserts, updates, and deletes. A current-state Silver table must interpret them using a reliable source sequence. Arrival time alone may be insufficient when messages are delayed. Delta Lake's merge documentation explains how merge operations apply updates and inserts, including the need to resolve multiple source rows that would ambiguously update one target row.

Plan for late events. Event time and ingestion time answer different questions. An order completed yesterday may arrive today and require yesterday's aggregate to change. In Spark Structured Streaming, watermarks govern late-data handling and state retention. They do not make delayed events disappear from the business; events beyond the supported window need a correction policy.

Publish freshness alongside the data. Consumers should be able to distinguish a successfully refreshed table from one whose upstream inputs remain delayed or incomplete.

For example, give a sales refresh a declared input cutoff. If orders have arrived through midnight but the customer feed is still several hours behind, decide whether to delay publication or publish with a visible completeness qualification. A run timestamp alone cannot explain that difference. Keep enough run metadata to connect a published result to its inputs and transformation version. When a correction requires a backfill, use that record to identify the dependent outputs that need rebuilding. This makes recovery a planned operation with a known scope instead of a manual search through every downstream report after an incident.

Medallion architecture example

Consider an illustrative equipment distributor that needs daily completed-order totals and fulfillment reporting. Its order system emits versioned changes, while its customer system supplies customer records. The following amounts and identifiers are invented to show the processing logic.

Bronze captures an order event for O-104, customer C-8, version 17, with status completed and an amount of USD 120. A delivery retry sends the same event again. Later, version 18 corrects the amount to USD 100. Retaining all three arrivals preserves evidence of both the duplicate delivery and the business correction.

Silver validates the required fields, parses the amount into a decimal, and matches C-8 to the customer dataset. It recognizes the repeated order version as a duplicate. The current-order table contains one row for O-104 at version 18, while a separate history table can preserve versions 17 and 18.

Gold produces a daily total grouped by completion date and currency. For this example, the documented measure sums the corrected amounts of orders currently marked completed; refunds are handled separately. The contribution from O-104 is USD 100, not USD 240 from the duplicate arrivals or USD 220 from summing both versions.

When version 18 arrives after the daily report has refreshed, the pipeline rebuilds the affected date's aggregate from the corrected Silver records. A different implementation could maintain the aggregate through change-aware updates, but simply appending another amount would be wrong.

The same Silver orders can feed fulfillment reporting without repeating ingestion and deduplication. If the customer match was wrong, engineers can correct that transformation and replay the retained inputs. The example's central design choice is separating delivery events, current business state, and reporting measures so that each has a clear meaning.

Benefits of medallion architecture

Shared preparation. A consistent Silver dataset gives multiple teams the same starting point. In the distributor example, sales and fulfillment reuse order identifiers and customer mappings. Improvements to those shared definitions can benefit both consumers, provided downstream changes are coordinated.

Targeted recovery. Preserved inputs make it possible to rerun a faulty transformation over the affected period. If a timestamp parser used the wrong time zone, the team can repair parsing before rebuilding dependent reports. This requires retaining the relevant source data, transformation code, and reference information.

More focused investigation. Layer boundaries help locate errors. An order missing from Bronze suggests a capture problem. An order present in Bronze but absent from accepted Silver records points toward validation or transformation. A correct Silver order with an incorrect Gold contribution points toward reporting logic.

Workload-specific preparation. A recurring dashboard can query a prepared aggregate, while exploratory analysis retains access to detailed records. Precomputation moves repeated work into the publishing pipeline; whether it improves overall cost depends on refresh frequency and reuse.

These benefits come from explicit interfaces between datasets. Teams gain the most when they can name the owner, accepted inputs, and intended consumers at each boundary.

Challenges of medallion architecture

Storage and processing overhead. Materialized layers retain multiple representations of related data. Their refresh jobs also consume compute. Table history adds another retention dimension: Delta Lake's utility documentation explains that removing obsolete files with VACUUM limits access to older table versions. Set replay and recovery requirements before choosing cleanup policies.

Accumulated latency. Frequent ingestion does not guarantee a fresh dashboard. A source delay, a slow Silver transformation, and an infrequent Gold refresh all contribute to end-to-end lag. Monitor the age of the business data reaching consumers, alongside job duration and success.

Shared-model disagreements. Different teams may use different definitions of an active customer or completed sale. Resolve common identifiers and basic validity rules centrally, but keep use-specific definitions explicit. Forcing every interpretation into one Silver table can make it difficult to reuse.

Schema changes and rejected records. A renamed field can break several dependent datasets. A permissive pipeline may keep running while quietly producing nulls. Databricks' expectations documentation illustrates distinct policies for invalid records, including retaining them with metrics, dropping them, or failing the update. Choose a policy for each rule and make discarded or quarantined data observable.

The maintenance burden grows with the number of datasets and dependencies. Every additional output should justify its refresh, ownership, and support costs through an actual consumer need.

How to implement medallion architecture

Start with one useful data product and build its path through the layers. For the distributor, that could be the daily completed-order report. A narrow implementation lets the team test recovery and correctness before expanding the pattern.

1. Define the output contract. Specify the report's grain, measures, currency treatment, time zone, freshness target, and owner. Decide how late corrections change previously published results. These requirements determine what upstream detail must survive.

2. Choose storage and processing boundaries. Select a supported table format, catalog, ingestion mechanism, and transformation engine. Decide whether layers need separate schemas or stronger administrative separation. Fabric's deployment guidance illustrates platform-specific choices; the same physical layout is not mandatory everywhere.

3. Capture replayable inputs. Preserve source identifiers, change ordering, ingestion metadata, and required payload fields. Record which inputs each run consumed. Test a restart after a write succeeds but the surrounding job reports failure, since that is where duplicate processing can become visible.

4. Build and test the shared model. Declare Silver keys and grain before writing joins. Test duplicate deliveries, missing references, source deletes, malformed values, and out-of-order updates. Reconcile accepted and rejected records with the inputs so that a successful job cannot conceal unexplained loss.

5. Publish and operate the output. Validate Gold totals against known cases. Track source lag, rejection rates, refresh completion, and downstream dependencies. Version transformation code and test backfills separately before replacing consumer-facing results. Assign access by need, with restricted handling of sensitive raw fields.

After the first product is reliable, add consumers that reuse its prepared records. Review whether each new output needs materialization or can be served through an existing query interface. Give each published dataset a catalog description with its grain, owner, refresh schedule, correction policy, and supported uses. Naming a table gold leaves all of those questions unanswered.

Detailed relationships may need a different query shape from the reporting aggregates. The distributor might want to trace orders through suppliers, components, and affected customers, provided those relationships are present in its tables. PuppyGraph lets teams define a graph schema over existing tables, mapping entities and relationships to nodes and edges. It connects to SQL databases, data warehouses, and data lakes or lakehouses, including direct reads of Iceberg and Delta Lake.

On its default direct-query path, the data stays in those sources, with no graph-specific ETL or persistent duplicate dataset required. Teams query the model using openCypher or Gremlin. PuppyGraph compiles queries into node and edge operators that run in its own distributed engine, enabling graph-specific traversal optimization. This adds a semantic view of the prepared data; the medallion pipelines still own ingestion, validation, and business corrections.

When should you use medallion architecture?

Medallion architecture is a useful fit when several consumers need different representations of the same source data. It is particularly valuable when source records are irregular, corrections are common, or engineers need to reconstruct outputs after transformation changes.

Assess the operational responsibilities before adopting all three layers. Can the team define a reusable Silver model? Is there a concrete Gold consumer? Can it retain enough inputs to support the promised recovery window? Are owners available to maintain shared transformations and coordinate changes?

A small reporting job over one stable, well-modeled source may need fewer persisted stages. Likewise, a request path that must respond synchronously to an operational event needs an appropriate serving design; routing it through scheduled analytical refreshes would miss that requirement.

Apply the pattern where intermediate datasets have a clear purpose. Its value increases when a shared boundary eliminates repeated preparation or makes failures easier to repair. Additional tables with no distinct responsibility mostly add work.

Conclusion

Medallion architecture separates source capture, reusable data preparation, and business-specific delivery. Bronze preserves inputs, Silver establishes reliable records, and Gold publishes interpretations for named consumers. The design becomes dependable through explicit keys, quality rules, correction policies, and observable freshness.

Start with one complete path from source to consumer, including a tested replay. Expand it when another workload can reuse the prepared data.

Try the forever-free PuppyGraph Developer Edition and book a demo with the team to see how openCypher and Gremlin queries traverse relationships across warehouse and lakehouse tables, with no graph-specific ETL, giving your curated data a graph query interface alongside its reporting uses.