Avro and Protobuf Schema Evolution

In an event-driven bank, a Kafka topic is a contract. The payments service publishes transfer events, and fraud monitoring, the ledger, customer notifications, regulatory reporting and the data platform all consume them, often owned by different teams and released on different schedules. A schema registry with compatibility checks is supposed to stop a producer from breaking those consumers. In practice, some of the most damaging contract breaks pass every automated check: the new schema is registered, CI is green, and a downstream consumer fails in production or, worse, keeps running and processes the data incorrectly. 

This article explains how compatibility modes really work in Confluent Schema Registry, how to govern schemas across teams, and the breaking-change scenarios that compatibility checks don't catch, with Avro as the main example and Protobuf differences where they matter. 

Compatibility modes: who upgrades first? 

Schema Registry checks each new schema version against earlier versions of the same subject, using the compatibility type configured for that subject. The modes are easiest to understand as an answer to one question: which side is deployed first, producers or consumers? 

BACKWARD (the default): a consumer using the new schema can read data written with the previous schema. This is safe when consumers upgrade first. In Avro, you can delete fields and add fields that have a default. 

FORWARD: a consumer using the previous schema can read data written with the new schema. This is safe when producers upgrade first. In Avro, you can add fields and delete fields that have a default. 

FULL: both directions hold, so either side can upgrade first. In Avro, this limits you to adding or removing fields that have defaults. 

NONE: no check at all. 

Each mode also has a transitive variant (BACKWARD_TRANSITIVE, FORWARD_TRANSITIVE and FULL_TRANSITIVE). The plain modes compare the new schema only with the latest version; the transitive modes compare it with every version ever registered for the subject. 

That direction question matters, because in banking the answer is usually the opposite of the default. A typical topic has one producing team and many consuming teams. The producer ships a new event version when its feature is ready; the consumers upgrade over weeks, on their own release trains. That is a producers-first world, which needs FORWARD or FULL compatibility. The default, BACKWARD, protects the opposite situation. 

For core financial events, we set FULL_TRANSITIVE explicitly on each subject: 

 1 curl -X PUT -H "Content-Type: application/vnd.schemaregistry.v1+json" \ 
 2   --data '{"compatibility": "FULL_TRANSITIVE"}' \ 
 3   https://schema-registry.internal/config/payments.transfer-events-value 

It is the most restrictive mode, and that's deliberate. Payment, ledger and account events are consumed widely, kept for a long time and replayed for reconciliation and reporting. 

A contract worth evolving 

Here is a transfer event in Avro, written with evolution in mind from day one: 

 1 { 
 2   "type": "record", 
 3   "name": "TransferEvent", 
 4   "namespace": "com.bank.payments.events", 
 5   "doc": "Emitted when a customer transfer changes state.", 
 6   "fields": [ 
 7     { "name": "transferId",  "type": "string" }, 
 8     { "name": "amountMinor", "type": "long", 
 9       "doc": "Amount in minor units of the currency, e.g. cents. Never negative." }, 
10     { "name": "currency",    "type": "string", "doc": "ISO 4217 alphabetic code." }, 
11     { "name": "status", 
12       "type": { "type": "enum", "name": "TransferStatus", 
13                 "symbols": ["UNKNOWN", "INITIATED", "SETTLED", "REJECTED"], 
14                 "default": "UNKNOWN" } }, 
15     { "name": "channel", "type": ["null", "string"], "default": null } 
16   ] 
17 } 

 Three habits are visible here: 

Units and meaning are documented in doc. Compatibility checks verify types, not meaning. The documentation is where meaning lives. 

The enum has an explicit fallback symbol and a default. As we'll see, enums are one of the most common sources of production breaks. 

Optional data is a nullable union with a null default, so its absence is explicit rather than replaced by an invented value. 

The breaking changes that pass CI 

Every scenario below is accepted by the registry under at least one common configuration, and every one can break, or silently corrupt, a downstream consumer. 

1. Checking the wrong direction 

An earlier version of the transfer event had a required debtorAccountId field with no default. The payments team removes it, because the new account model publishes it in a separate event. Under the default BACKWARD mode, deleting a field is allowed: a consumer using the new schema doesn't need it, so the registry accepts the change. 

The producer deploys first, as it usually does. Consumers still on the old schema, such as fraud monitoring, now receive events without a field their schema declares as required. With no default to fall back on, Avro can't fill the gap and deserialisation fails. The check passed because it protected the wrong direction. The fix is the mode choice above: if producers deploy first, use FORWARD or FULL, which only allow removing fields that have a default. 

2. Adding an enum symbol 

The business introduces a new status, PENDING_VERIFICATION, for Verification of Payee checks. The producer adds the symbol to TransferStatus. Under BACKWARD, this is fine: a new reader understands every old symbol. 

But a consumer still on the old schema receives PENDING_VERIFICATION, a symbol it doesn't know. Without an enum default in the reader's schema, Avro can't resolve it and deserialisation fails. In a producers-first world, this is one of the most frequent production incidents we see. Two defences help: 

Give every enum a fallback symbol and a default from its first version, as in the schema above, so old readers map unknown values to UNKNOWN instead of failing. 

Make consumers handle UNKNOWN deliberately: park the event, alert, and don't guess. 

Protobuf behaves differently. Proto3 enums are open, so an unknown value is kept as its number, and Java code sees it as UNRECOGNIZED. Deserialisation doesn't fail, but a switch statement without a default branch will silently ignore the new status. The failure moves from the deserialiser to business logic, which is harder to detect. 

3. A default value that isn't true 

The payments platform was euro-only, and events had no currency field. When RON accounts are added, the team adds currency with a default of "EUR". The schema is fully compatible, and every check passes. 

When a consumer reads old events with the new schema, Avro fills in the default, so every historical event now says EUR. That is correct for the old data. Then a second producer, a legacy adapter for a newly integrated entity that still publishes with the old schema, starts sending RON transfers without the field. Every one of them is read as EUR. 

A default is a value the reader will believe. It is only safe if it is true for every event that could arrive without the field, now and in the future. When there is any doubt, use a nullable field and make consumers handle "missing" explicitly. 

4. The same type with a different meaning 

An amount field of type long held whole currency units. A developer changes the producer to publish minor units (cents), because that's how the new core system works, and updates the doc string. The schema type hasn't changed, so the registry has nothing to check, and consumers start reading amounts that are a hundred times too large. 

No compatibility mode can catch a semantic change. The protection is process: a change of meaning is a breaking change and gets a new field name. Here that means adding amountMinor and deprecating amount over an agreed period, and never reusing a field name with a new meaning. 

The Protobuf version of this mistake is reusing a field number. Removing field 4 and later adding a new field with number 4 makes old data decode into the new field. Protobuf has a mechanism for exactly this: 

 1 message TransferEvent { 
 2   reserved 4; 
 3   reserved "fee"; 
 4  
 5   string transfer_id   = 1; 
 6   int64  amount_minor  = 2; 
 7   string currency      = 3; 
 8   TransferStatus status = 5; 
 9 } 

 Every removed field number and name goes into reserved, so it can never be reused by accident. 

5. Renaming a field 

Avro supports field aliases, so a reader schema can rename customerId to partyId and still read old data. Protobuf goes further: names aren't on the wire at all, so renaming a field is invisible to the binary format. 

Both are "compatible", and both can break consumers that don't use generated classes: generic-record processors, stream-processing SQL, connectors writing to a data lake, and anything that converts events to JSON. In those systems, a rename means a column disappears and a new one appears. In banking, the data platform feeding regulatory reports is often exactly that kind of consumer. Treat renames as breaking changes unless every consumer has confirmed otherwise. 

6. Compatible with the last version, not with the first 

Under a non-transitive mode, version 3 is checked only against version 2. If version 2 relaxed something that version 1 relied on, version 3 can be incompatible with version 1, and the registry won't notice. 

This matters in banking because topics are replayed. A compacted account-state topic, a reconciliation job reprocessing a quarter, or a new consumer reading from the earliest offset will meet version 1 events. A consumer on version 3 then fails on data the registry considered history. Use the transitive modes for any topic that can be replayed, which in practice is most financial topics. 

7. Checking against the wrong history 

CI runs a compatibility check and passes, but against a development registry whose subject history differs from production. Or the producer switches subject naming strategy (for example from the default TopicNameStrategy to RecordNameStrategy), or changes the record namespace, and the new schema lands in a new, empty subject. A new subject has no history, so any schema is compatible with it. 

Both cases are invisible until a production consumer fails. The fixes are to run CI compatibility checks against the production registry (read-only), and to treat any change of subject name as a breaking change that needs review. 

Governance that holds across teams 

Compatibility modes are a tool. Governance is what makes them effective in an organisation with many producing and consuming teams. 

Schemas are code, owned by the producer. Each event schema lives in the producing team's repository (or a shared contracts repository) with clear ownership. Consuming teams are added as required reviewers on changes to the schemas they depend on. 

Only the pipeline registers schemas in production. Producers in production run with automatic registration switched off, so an application can never register a schema on its own: 

 1 spring: 
 2   kafka: 
 3     producer: 
 4       value-serializer: io.confluent.kafka.serializers.KafkaAvroSerializer 
 5       properties: 
 6         schema.registry.url: https://schema-registry.internal 
 7         auto.register.schemas: false 
 8         use.latest.version: true 
 9     consumer: 
10       value-deserializer: io.confluent.kafka.serializers.KafkaAvroDeserializer 
11       properties: 
12         schema.registry.url: https://schema-registry.internal 
13         specific.avro.reader: true 

 Write access to the production registry is limited to the deployment pipeline, with the same change controls as any other production change. 

CI checks compatibility against production. The Schema Registry Maven plugin tests a schema against the subject's real history before anything is merged: 

 1  
 2   io.confluent 
 3   kafka-schema-registry-maven-plugin 
 4   ${confluent.version} 
 5    
 6      
 7       ${prod.schema.registry.readonly.url} 
 8      
 9      
10       src/main/avro/TransferEvent.avsc 
11      
12    
13    
14      
15       verify 
16       test-compatibility 
17      
18    
19  

Consumers test the producer's examples. Compatibility checks can't catch scenarios 3 to 5, because they are about meaning. Consumer-side contract tests can. The producer publishes a set of example events for each schema version, including edge cases such as new enum values, missing optional fields and large amounts. Each consumer runs those examples through its real deserialisation and business logic in its own CI. A new example event that a consumer can't handle fails that consumer's build before the producer releases. 

Breaking changes get a new topic. When a change truly can't be made compatibly, we don't force it through with NONE. The producer publishes to a new versioned topic (for example payments.transfer-events.v2) alongside the old one, consumers migrate on their own schedule, and the old topic is retired after an agreed deprecation period. It costs more for a few weeks, but nobody is surprised in production. 

Rules beyond types. Confluent's data contracts extend the registry with metadata and rules, such as tagging fields as personal data or validating that an amount is never negative. They are a useful addition for teams that want some semantic checks enforced at the producer, alongside the practices above. 

A short checklist for every schema change 

Before approving a change to a financial event schema, we ask: 

• Which side deploys first, and does the subject's compatibility mode protect that direction? 
• Is the mode transitive, given that the topic may be replayed from the beginning? 
• Does any new default describe something that is true for every event that could arrive without the field? 
• Has any field changed meaning, unit or precision while keeping its name, or (in Protobuf) its number? 
• Were enum values added, and does every consumer handle an unknown value deliberately? 
• Is any field renamed, and do any consumers read fields by name? 
• Did CI check the schema against the production subject, under the same subject name? 

Closing thought 

A schema registry is the safety net for event contracts in a bank, but it only checks what it can see: types, defaults and field structure. The changes that cause real incidents are the ones it can't see, such as the wrong compatibility direction, a default that isn't true, a field whose meaning changed and a new subject with no history. Choosing compatibility modes deliberately, registering schemas only through the pipeline and having consumers test the producer's examples turn the registry from a formality into a contract both sides can rely on.