What Is Avro? Format & Architecture

Apache Avro solves a narrow problem that shows up constantly in distributed systems: a program writes a record today, and another program, maybe in a different language, maybe deployed weeks later, has to read that same record correctly. That reliability requirement, not a preference for one wire format over another, is why Avro became Confluent Schema Registry's original default format for Kafka and a mainstay of Hadoop-ecosystem pipelines, and why it still underlies part of the modern data lakehouse today.

This post covers Avro's architecture and file format, its core features, how its schemas are structured, how serialization and deserialization actually happen, the rules that govern schema evolution, and where Avro's row-based design is the right tool versus where a columnar format like Parquet or ORC, or a text format like JSON, fits better.

What is Apache Avro?

Apache Avro is a data serialization system: a specification for defining record schemas in JSON, encoding data written against those schemas into a compact binary (or JSON) form, and resolving differences between the schema a record was written with and the schema a reader expects to see. The Avro project calls itself "the leading serialization format for record data, and first choice for streaming data pipelines," reflecting design choices shaped specifically by event-driven and distributed-processing workloads rather than general-purpose object serialization.

Doug Cutting, who had already created Hadoop, proposed Avro as a new Hadoop subproject in April 2009, writing that his "ambition is for Avro to replace both Hadoop's RPC and to be used for most Hadoop data files, e.g., by Pig, Hive, etc." (Apache Hadoop mailing list, April 2009), a fix for the fact that Hadoop's existing Writable serialization was Java-only and could not move data cleanly between the ecosystem's other, non-Java tools. It graduated to a top-level Apache project in May 2010 (ASF announcement). The project borrows its name, and its original logo, from A.V. Roe and Company, the British aircraft manufacturer.

The design decision that shapes everything else in this post: Avro carries the schema alongside the data, or makes it retrievable by reference, and resolves any difference between writer and reader schemas at read time rather than through generated code or fixed field numbers baked into the wire format. That choice is why Avro files are self-describing, why schema evolution has formal rules instead of being an afterthought, and why Avro fit naturally into Hadoop's dynamically typed, multi-language environment from the start.

Apache Avro architecture

Avro data appears on disk or on the wire in one of a few concrete shapes. The most common at-rest shape is the object container file: a self-contained binary file that carries its own schema plus zero or more blocks of serialized records. The file specification defines the header as three parts: four magic bytes ('O', 'b', 'j', followed by the byte 1), a metadata map that must include avro.schema (the writer's schema, as JSON) and, when compression is used, avro.codec, and a 16-byte randomly generated sync marker unique to the file. Required codecs are null (uncompressed) and deflate; bzip2, snappy, xz, and zstandard are optional depending on the implementation. After the header, the file is a sequence of blocks, each holding a count of objects, the byte size of the serialized data, the serialized objects themselves, and a repeat of the sync marker, which lets a reader resynchronize after a corrupted or truncated block and lets a splitting framework divide a large file at block boundaries without parsing every record.

Because Avro's binary encoding itself carries no field names, tags, or type markers, a container file is self-describing but the encoded bytes are not: decoding a record correctly requires the exact schema it was written with, whether that comes from the file's own header, an out-of-band schema, or a schema registry (covered below). That dependency on an available writer schema, rather than the encoding carrying its own field identifiers, is the trade Avro makes for a smaller wire format, and it is what makes schema resolution, not just schema presence, a first-class part of the specification.

Key features of Apache Avro

Compact binary encoding. Because field names and types live in the schema rather than in the encoded bytes, Avro's binary form has none of the repeated key overhead that a self-describing text format like JSON carries on every record, which is a meaningful difference at the row counts a streaming platform like Kafka pushes through in a day.

Rich, schema-defined data structures. Avro schemas are not limited to flat key-value records. The specification's complex types cover records (named, ordered fields), enums (a fixed set of named symbols), arrays, maps (keyed by string), unions (a value that may be one of several types, commonly used to express an optional field as a union with null), and fixed (binary data of a declared, constant length). Records can nest arbitrarily, so Avro models the same nested structures JSON does, with a schema enforcing what shapes are valid.

Schema evolution as a specified behavior, not a convention. The specification defines exactly how a reader schema and a writer schema are reconciled when they differ, covering added fields, removed fields, renamed fields, and type promotion. The full rules are covered in the schema evolution section below; the feature worth noting here is that this reconciliation is part of the Avro specification itself, so any conforming implementation behaves the same way.

Dynamic use without required code generation. Because the schema always travels with, or is resolvable alongside, the data, a program can read and write Avro records generically, as a map of field names to values, without first generating a language-specific class from the schema. Code generation is available and commonly used for the JVM and Python, where it produces typed classes for convenience, but it is an option rather than a prerequisite, which matters for tools that process arbitrary Avro data without knowing its schema ahead of time. The specification also defines a protocol wire format for remote procedure calls that reuses the same schema and encoding machinery, though in practice Avro's footprint today is overwhelmingly in data serialization rather than RPC.

Every one of these features traces back to the same design goal: a producer and a consumer that were never compiled together, and may never run the same version of anything, still need to agree on what a record means. Compact encoding, rich structure, specified evolution, and optional code generation are four different answers to that one requirement.

Understanding Avro schemas

An Avro schema is itself a JSON document, which is what lets Avro schemas be versioned, diffed, and stored in ordinary text-based systems like git or a schema registry. A record schema declares a type of "record", a name, an optional namespace for qualifying that name, and a fields array, where each field has a name, a type, and optionally a doc, a default value, an order for sort comparisons, and aliases for alternate names.

{
  "type": "record",
  "name": "User",
  "namespace": "com.example.accounts",
  "fields": [
    {"name": "id", "type": "long"},
    {"name": "username", "type": "string"},
    {"name": "email", "type": ["null", "string"], "default": null},
    {"name": "signupTimestamp", "type": {"type": "long", "logicalType": "timestamp-millis"}}
  ]
}

The eight primitive types are null, boolean, int (32-bit), long (64-bit), float, double, bytes, and string. The email field above shows the common pattern for an optional value: a union of null and string, with a default of null, which is also what makes the field safe to add to an existing schema without breaking readers written against the older version (more in the schema evolution section). The signupTimestamp field shows a logical type: timestamp-millis layers a semantic interpretation, milliseconds since the Unix epoch, on top of the physical long encoding, so the wire format is unchanged but tooling knows to treat the value as a timestamp.

Teams that find hand-written schema JSON hard to read can author schemas in Avro IDL instead, a more compact syntax closer to Java, C++, or Python than to JSON, that compiles down to the same schema. Both are equivalent; IDL exists purely for authoring convenience, since the JSON form is what implementations actually consume and exchange.

What all of this is for is portability: because a schema is a plain JSON document rather than a compiled artifact, it can sit in a container file's header, live in a registry, or travel in a git diff, and still describe the data precisely enough for another program, in another language, to trust it.

Avro serialization and deserialization

Avro defines two ways to encode a value against its schema: a compact binary encoding, used almost everywhere in practice, and a far more verbose JSON encoding, useful mainly for debugging or for systems that already move JSON around and want Avro's schema validation without adopting the binary wire format.

Because Avro's binary bytes carry no field identifiers, every serialization and deserialization call is really a pair: a writer schema, the schema a producer used to encode a given record, and a reader schema, the schema the consumer wants the data back as. When the two are identical, decoding is a direct walk through the schema. When they differ, in a way compatible with the schema evolution rules below, Avro resolves them at read time: a reader is constructed with both schemas and produces the reader's shape from the writer's bytes.

Where that writer schema comes from depends on the deployment. An object container file embeds it in the header, so the file is self-sufficient. A Kafka topic backed by Avro typically does not repeat the schema on every message, since that would erase Avro's size advantage; instead, a system like Confluent's Schema Registry stores each schema version once and has producers embed a compact schema ID in each message, which consumers use to fetch the matching writer schema before decoding.

A minimal example using Python's fastavro library shows the container-file path end to end:

import fastavro

schema = {
    "type": "record",
    "name": "User",
    "fields": [
        {"name": "id", "type": "long"},
        {"name": "username", "type": "string"},
        {"name": "email", "type": ["null", "string"], "default": None},
    ],
}

records = [{"id": 1, "username": "avery", "email": None}]

with open("users.avro", "wb") as out:
    fastavro.writer(out, schema, records)

with open("users.avro", "rb") as f:
    for record in fastavro.reader(f):
        print(record)

fastavro.writer writes the schema into the file header once and then the records in binary form; fastavro.reader reads that header back and yields records already resolved against it, without the caller supplying a schema at all, since the container file already has one.

Schema evolution in Apache Avro

Schema evolution is the problem Avro is most often chosen to solve: producers and consumers deploy independently, so the schema a message was written with and the schema a consumer currently expects will diverge, and the specification defines exactly what happens when they do.

Consider a schema that adds a field:

// v1 (writer, older)
{"type": "record", "name": "User", "fields": [
  {"name": "id", "type": "long"},
  {"name": "username", "type": "string"}
]}


// v2 (reader, newer)
{"type": "record", "name": "User", "fields": [
  {"name": "id", "type": "long"},
  {"name": "username", "type": "string"},
  {"name": "email", "type": ["null", "string"], "default": null}
]}

A consumer reading v1 data with the v2 schema gets email: null for every record, because the specification states that "if the reader's record schema has a field that contains a default value, and writer's schema does not have a field with the same name, then the reader should use the default value from its field." The reverse direction works too: a consumer still running the v1 reader schema against v2 data simply ignores the email field it does not know about, since "if the writer's record contains a field with a name not present in the reader's record, the writer's value for that field is ignored." A field can only be safely added or removed without breaking the other side if it carries a default; a required field with no default that is missing from the writer's data is an error, not a null.

Two other resolution rules matter in practice. Type promotion lets narrower numeric types widen automatically: an int value can be read as long, float, or double; a long as float or double; and a float as double; so widening a field's declared type is compatible without a default. Aliases let a field or a named type be renamed across versions: the reader schema lists the old name as an alias, and resolution matches on either the current name or an alias, which is how a schema can fix an early naming mistake without breaking every existing reader.

None of these rules require a central authority; they are properties of any two schemas an implementation resolves against each other. In practice, most teams add one on top: a registry that assigns a compatibility mode to every schema change before allowing it. Confluent's Schema Registry, for example, checks each new schema against a configured mode, backward, forward, or full, where backward compatibility, its default, requires a consumer using the new schema to be able to read data already written under the old one, in practice meaning new fields need defaults so the new-schema reader can fill them in on old data that lacks them, while fields can be dropped freely. That check enforces the same underlying Avro resolution rules, applied before a producer is allowed to publish.

Advantages and limitations of Apache Avro

Advantages. Specified schema evolution is the main reason Avro remains Confluent Schema Registry's default format for Kafka: producers and consumers deploy on independent schedules, and Avro defines in advance what a version mismatch does rather than leaving it to fail at runtime. Its binary encoding is compact because field names live in the schema rather than in every record. It requires no code generation to read or write data generically, which matters for schema registries and generic ETL tools that have to handle Avro data without knowing its shape ahead of time. And its container files are self-describing, which makes an Avro file portable in a way a headerless binary format is not.

Limitations. Avro is row-based: a record's fields are stored together, efficient for writing and reading whole records back out, but inefficient for a query that only needs a handful of columns from a wide record, since a row-oriented reader still has to deserialize the fields it does not need to get to the ones it does. It offers no native column pruning or the columnar compression ratios that formats built for analytical scanning achieve. And unlike JSON, the encoded bytes are not human-readable without the schema and a decoder, a real cost during debugging even though it is the same design choice that keeps the format compact.

That trade-off, row-oriented and schema-driven versus columnar and scan-optimized, is most visible next to the formats Avro is usually mentioned alongside:

The row for schema handling explains the rest of the table: because Parquet and ORC embed their schema in the footer of each file, a query engine that only needs three columns out of eighty can skip straight to reading those columns' data, which is what makes them fit for warehouse-scale analytical scans. Avro's row-based layout does not offer that shortcut, so it is not the format teams reach for when the workload is scanning a billion rows to aggregate four columns. It is the format teams reach for when the workload is getting a just-produced record read correctly by something downstream, possibly written in a different language, possibly deployed after the producer's next schema change. A common architecture uses both: Avro for the stream of individual events as they happen, and Parquet for the lake once that data is compacted into large, column-oriented files, each format doing the job it is built for.

One further fact belongs here even though it is neither an advantage nor a limitation of Avro itself: Avro's reach extends into the lakehouse's metadata layer in a way that has nothing to do with event streaming. Apache Iceberg's table specification defines the manifest files that track a snapshot's data files as Avro outright, and every Iceberg implementation writes the manifest lists that reference those manifests as Avro too, regardless of whether the table's actual data files are Parquet, ORC, or Avro. Reading an Iceberg table, in other words, already means reading through an Avro-encoded metadata layer to find the current data files, even on a table whose rows never touch Avro at all. PuppyGraph connects to Iceberg tables directly, resolving that metadata layer to read the underlying data files in place rather than through a separate export step. A user-defined schema maps the resulting rows to nodes and edges, and openCypher and Gremlin queries run against them in PuppyGraph's own distributed engine, whatever format the table's data files are stored in underneath.

Conclusion

Avro's core idea is narrow and specific: keep the schema separate from, or referenced by, the data, encode records compactly because the schema does not need to repeat, and define exactly what happens when a reader's schema and a writer's schema disagree. That idea made it the natural fit for Kafka topics and Hadoop pipelines, where producers and consumers change independently and a version mismatch has to be handled by rule rather than by luck. It is not the right tool for large analytical scans, where a columnar format's ability to read only the columns a query touches wins, and it is not the right tool for a human-authored document, where JSON's plain text wins. Choosing among Avro, Parquet, ORC, and JSON is really a question about what happens to the data next: whether it is a record being handed to the next process in a pipeline, a table being scanned by a query engine, or a document a person needs to read.

Try the forever-free PuppyGraph Developer Edition and book a demo with the team to see how openCypher and Gremlin queries run over warehouse and lakehouse tables, with no graph-specific ETL, whatever file format sits underneath them, Avro, Parquet, or ORC.