Exactly-Once Semantics in Kafka Payment Pipelines
Transactional Producers and Consumers
"Kafka guarantees exactly-once" is the sentence that gets a payments team into trouble, because it's true in exactly the narrow sense that matters least to the people repeating it. Kafka can guarantee exactly-once processing for a read-process-write sequence entirely within its own boundary. The moment that sequence touches a database — which a payment pipeline always does — the guarantee needs help it doesn't come with automatically. This is a walkthrough of what Kafka's transactional API actually does, what it doesn't, and where the seam is that a payments team needs to close itself.
Why "at-least-once" is the honest default, and why it produces real duplicates
A Kafka producer, configured with nothing beyond the basics, gives you at-least-once delivery. A broker acknowledgment gets lost on the way back to the producer, the producer retries, and the broker — having already written the first attempt — now has two copies of the same logical event. A consumer has the mirror problem: it processes a batch, crashes or gets rebalanced before committing its offset, and the next consumer instance picks up from the last committed offset, reprocessing records that were already handled.
For a payment-initiated event flowing through a pipeline that eventually calls a settlement API or writes a ledger entry, either of those duplicates is a double-charge or a double-posted transaction, not a cosmetic log artifact. This is the exact failure class idempotency keys exist to catch at the API boundary — we've covered that pattern in detail elsewhere. What this piece is about is closing the same gap one layer earlier, inside the pipeline itself, so the duplicate never reaches the point where an idempotency key has to catch it.
Idempotent producers solve one specific problem, not the general one
Setting enable.idempotence=true (the default in modern Kafka clients) assigns the producer a Producer ID and attaches a monotonically increasing sequence number to every message sent to a given partition. The broker deduplicates on (PID, sequence number), so a producer retry after a lost acknowledgment doesn't create a second copy — the broker recognizes the sequence number it already has and discards the retry.
That's a real guarantee, and it's necessary. It's also narrow: it protects against producer-side retries to a single partition. It says nothing about a consumer reprocessing a batch after a crash, and nothing about atomicity across multiple partitions or topics — which is exactly what a payment pipeline usually needs, because "consume the payment-initiated event, write the ledger entry, produce the payment-settled event" touches more than one topic.
Transactions: atomicity across the whole read-process-write sequence
Kafka's transactional API is what closes the rest of the gap. Configuring a transactional.id on the producer lets it group multiple sends — potentially across several topics and partitions — into a single atomic unit, and critically, lets it include the consumer's offset commit inside that same transaction. Either everything in the transaction becomes visible to downstream consumers, or none of it does.
1 @Configuration
2 public class PaymentKafkaConfig {
3
4 @Bean
5 public ProducerFactory producerFactory() {
6 Map props = new HashMap<>();
7 props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, bootstrapServers);
8 props.put(ProducerConfig.ENABLE_IDEMPOTENCE_CONFIG, true);
9 props.put(ProducerConfig.ACKS_CONFIG, "all");
10 // Unique per instance — Spring Boot appends an instance suffix
11 // automatically when you set transaction-id-prefix.
12 props.put(ProducerConfig.TRANSACTIONAL_ID_CONFIG, "payment-pipeline-tx-");
13 DefaultKafkaProducerFactory factory =
14 new DefaultKafkaProducerFactory<>(props);
15 factory.setTransactionIdPrefix("payment-pipeline-tx-");
16 return factory;
17 }
18
19 @Bean
20 public KafkaTransactionManager kafkaTransactionManager(
21 ProducerFactory producerFactory) {
22 return new KafkaTransactionManager<>(producerFactory);
23 }
24 } With Spring Boot, most of this collapses to a single property — spring.kafka.producer.transaction-id-prefix — and Spring auto-configures the KafkaTransactionManager and wires it into the listener container. The transactional.id has to be unique per producer instance across your fleet; reusing it across instances is what causes a producer to fence itself off, which is the mechanism working correctly, not a bug, but confusing the first time it happens in production.
The consumer side: committing offsets inside the transaction, not after it
This is the part that's easy to get wrong even with transactions enabled: committing the consumer offset as a separate step after the transaction commits reopens the exact gap transactions exist to close. If the process crashes between the Kafka transaction commit and the offset commit, the next consumer instance reprocesses the same batch — the transaction protected the write, but not the read-process-write sequence as a whole.
The fix is producer.sendOffsetsToTransaction(), which folds the offset commit into the same atomic transaction as the outbound sends:
1 @Service
2 public class PaymentEventProcessor {
3
4 @Transactional("kafkaTransactionManager")
5 @KafkaListener(topics = "payment-initiated", groupId = "settlement-service")
6 public void handlePaymentInitiated(PaymentInitiatedEvent event) {
7 LedgerEntry entry = ledgerService.postPending(event); // see caveat below
8 PaymentSettledEvent settled = settlementService.process(event, entry);
9
10 kafkaTemplate.send("payment-settled", settled.paymentId(), settled);
11 // Spring's listener container calls sendOffsetsToTransaction()
12 // automatically here, as part of the same transaction, when the
13 // container's ackMode is set to RECORD and a KafkaTransactionManager
14 // is configured — this is what actually closes the gap.
15 }
16 } The @Transactional("kafkaTransactionManager") annotation, combined with the listener container being configured with a KafkaAwareTransactionManager, is what makes Spring send the consumed offset and the produced event as one atomic operation. If handlePaymentInitiated throws, the container rolls the consumer back to the un-acknowledged offset — the record gets redelivered — and none of the partial output becomes visible downstream, because the transaction that would have made it visible never committed.
Downstream consumers need to opt in, explicitly
Kafka's transactional guarantee only protects consumers that ask for it. A downstream consumer reading payment-settled with the default isolation.level=read_uncommitted will see uncommitted, in-flight transactional writes — including ones that later get aborted — which defeats the entire point. Every consumer downstream of a transactional producer needs:
1 spring:
2 kafka:
3 consumer:
4 properties:
5 isolation.level: read_committed
6 This is the setting that actually determines whether "exactly-once" reaches the next service in the chain or quietly stops at the first hop. It's also the setting that's easiest to forget, because a consumer that's missing it doesn't error — it just silently sees data it shouldn't, including phantom records from a transaction that was rolled back.
Where the guarantee actually ends: the database is a separate resource
This is the caveat that matters most for a payment pipeline specifically, and it's the one "Kafka gives you exactly-once" glosses over. Kafka's transactional guarantee covers Kafka's own topics and consumer offsets. It does not extend to a database write in the same method, because the database and Kafka are two independent transactional resources with no shared coordinator. In the code above, ledgerService.postPending(event) runs in a normal JDBC or JPA transaction, entirely separate from the Kafka transaction wrapping the method. If the ledger write commits and the process crashes before the Kafka transaction commits, you have a ledger entry with no corresponding settled event. If it happens the other way — Kafka commits, the database write fails — you have a settled event with no ledger record behind it.
Spring does offer a ChainedKafkaTransactionManager to synchronize a JDBC transaction manager with a Kafka one, but it's been deprecated since Spring Kafka 2.7, for a reason worth taking seriously rather than working around: chaining two independent transaction managers isn't real atomicity, it's sequencing — one commits, then the other does, and a crash in the gap between the two still leaves you inconsistent. True cross-resource atomicity would require distributed transactions (XA), which Kafka doesn't support natively and which carries enough of its own performance and complexity cost that it's rarely the right trade for a payment pipeline anyway.
The pattern that actually holds up is the transactional outbox: write the ledger entry and a corresponding outbox row in the same database transaction — a single resource, genuinely atomic — then have a separate process (Debezium's CDC connector against the outbox table is the common choice) publish the outbox row to Kafka asynchronously, with its own retry and idempotent-publish logic. That process reintroduces at-least-once delivery for the outbound event, which is exactly why the idempotency-key pattern at the consuming service's API boundary still matters even after all of this is correctly wired — Kafka transactions eliminate duplicates within the pipeline; they don't remove the need for the receiving service to be able to safely process the same event twice if the outbox publisher retries.
What this costs, and when it's worth paying
Worth saying plainly for whoever's signing off on the architecture, not just the engineer implementing it: transactional writes carry real overhead — coordinator round-trips, read_committed consumers buffering until a transaction resolves, and Kafka 4.0's KRaft-mode improvements narrowing but not eliminating that gap. For a high-volume, latency-sensitive stream where a duplicate is genuinely harmless — a metrics pipeline, an audit log with its own downstream dedup — at-least-once with idempotent consumers is usually the better trade. For a payment-initiated-to-settled pipeline, where a duplicate is a double-charge, the latency cost of transactions is the cheaper problem to have.
The shape of a pipeline that actually holds
Put together, a payment pipeline that earns the phrase "exactly-once" end to end looks like this: idempotent, transactional producers with a stable transactional.id per instance; consumer offset commits folded into the same transaction via sendOffsetsToTransaction, which Spring wires up automatically behind @Transactional; every downstream consumer explicitly set to read_committed; the database write kept in its own atomic transaction alongside an outbox row rather than chained to Kafka's transaction manager; and an idempotency key at the API boundary of whatever service consumes the eventual outbound event, because the outbox publisher itself is back to at-least-once delivery by design. None of these pieces individually claims to solve the whole problem. Wired together deliberately, with the seams between them understood rather than assumed away, they do.
