"Kafka guarantees exactly-once" is the sentence that gets a payments team into trouble, because it's true in exactly the narrow sense that matters least to the people repeating it. Kafka can guarantee exactly-once processing for a read-process-write sequence entirely within its own boundary. The moment that sequence touches a database — which a payment pipeline always does — the guarantee needs help it doesn't come with automatically. This is a walkthrough of what Kafka's transactional API actually does, what it doesn't, and where the seam is that a payments team needs to close itself. 

Why "at-least-once" is the honest default, and why it produces real duplicates 

A Kafka producer, configured with nothing beyond the basics, gives you at-least-once delivery. A broker acknowledgment gets lost on the way back to the producer, the producer retries, and the broker — having already written the first attempt — now has two copies of the same logical event. A consumer has the mirror problem: it processes a batch, crashes or gets rebalanced before committing its offset, and the next consumer instance picks up from the last committed offset, reprocessing records that were already handled. 

For a payment-initiated event flowing through a pipeline that eventually calls a settlement API or writes a ledger entry, either of those duplicates is a double-charge or a double-posted transaction, not a cosmetic log artifact. This is the exact failure class idempotency keys exist to catch at the API boundary — we've covered that pattern in detail elsewhere. What this piece is about is closing the same gap one layer earlier, inside the pipeline itself, so the duplicate never reaches the point where an idempotency key has to catch it. 

Idempotent producers solve one specific problem, not the general one 

Setting enable.idempotence=true (the default in modern Kafka clients) assigns the producer a Producer ID and attaches a monotonically increasing sequence number to every message sent to a given partition. The broker deduplicates on (PID, sequence number), so a producer retry after a lost acknowledgment doesn't create a second copy — the broker recognizes the sequence number it already has and discards the retry. 

That's a real guarantee, and it's necessary. It's also narrow: it protects against producer-side retries to a single partition. It says nothing about a consumer reprocessing a batch after a crash, and nothing about atomicity across multiple partitions or topics — which is exactly what a payment pipeline usually needs, because "consume the payment-initiated event, write the ledger entry, produce the payment-settled event" touches more than one topic. 

Transactions: atomicity across the whole read-process-write sequence 

Kafka's transactional API is what closes the rest of the gap. Configuring a transactional.id on the producer lets it group multiple sends — potentially across several topics and partitions — into a single atomic unit, and critically, lets it include the consumer's offset commit inside that same transaction. Either everything in the transaction becomes visible to downstream consumers, or none of it does. 

    public ProducerFactory<String, PaymentEvent> producerFactory() { 

        Map<String, Object> props = new HashMap<>(); 

        props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, bootstrapServers); 

        props.put(ProducerConfig.ENABLE_IDEMPOTENCE_CONFIG, true); 

        props.put(ProducerConfig.ACKS_CONFIG, "all"); 

        // Unique per instance — Spring Boot appends an instance suffix 

        // automatically when you set transaction-id-prefix. 

        props.put(ProducerConfig.TRANSACTIONAL_ID_CONFIG, "payment-pipeline-tx-"); 

        DefaultKafkaProducerFactory<String, PaymentEvent> factory = 

            new DefaultKafkaProducerFactory<>(props); 

        factory.setTransactionIdPrefix("payment-pipeline-tx-"); 

    public KafkaTransactionManager<String, PaymentEvent> kafkaTransactionManager( 

            ProducerFactory<String, PaymentEvent> producerFactory) { 

        return new KafkaTransactionManager<>(producerFactory); 

 With Spring Boot, most of this collapses to a single property — spring.kafka.producer.transaction-id-prefix — and Spring auto-configures the KafkaTransactionManager and wires it into the listener container. The transactional.id has to be unique per producer instance across your fleet; reusing it across instances is what causes a producer to fence itself off, which is the mechanism working correctly, not a bug, but confusing the first time it happens in production. 

The consumer side: committing offsets inside the transaction, not after it 

This is the part that's easy to get wrong even with transactions enabled: committing the consumer offset as a separate step after the transaction commits reopens the exact gap transactions exist to close. If the process crashes between the Kafka transaction commit and the offset commit, the next consumer instance reprocesses the same batch — the transaction protected the write, but not the read-process-write sequence as a whole. 

The fix is producer.sendOffsetsToTransaction(), which folds the offset commit into the same atomic transaction as the outbound sends: 

    @Transactional("kafkaTransactionManager") 

    @KafkaListener(topics = "payment-initiated", groupId = "settlement-service") 

    public void handlePaymentInitiated(PaymentInitiatedEvent event) { 

        LedgerEntry entry = ledgerService.postPending(event); // see caveat below 

        PaymentSettledEvent settled = settlementService.process(event, entry); 

        kafkaTemplate.send("payment-settled", settled.paymentId(), settled); 

        // Spring's listener container calls sendOffsetsToTransaction() 

        // automatically here, as part of the same transaction, when the 

        // container's ackMode is set to RECORD and a KafkaTransactionManager 

        // is configured — this is what actually closes the gap. 

 The @Transactional("kafkaTransactionManager") annotation, combined with the listener container being configured with a KafkaAwareTransactionManager, is what makes Spring send the consumed offset and the produced event as one atomic operation. If handlePaymentInitiated throws, the container rolls the consumer back to the un-acknowledged offset — the record gets redelivered — and none of the partial output becomes visible downstream, because the transaction that would have made it visible never committed. 

Downstream consumers need to opt in, explicitly 

Kafka's transactional guarantee only protects consumers that ask for it. A downstream consumer reading payment-settled with the default isolation.level=read_uncommitted will see uncommitted, in-flight transactional writes — including ones that later get aborted — which defeats the entire point. Every consumer downstream of a transactional producer needs: 

This is the setting that actually determines whether "exactly-once" reaches the next service in the chain or quietly stops at the first hop. It's also the setting that's easiest to forget, because a consumer that's missing it doesn't error — it just silently sees data it shouldn't, including phantom records from a transaction that was rolled back. 

Where the guarantee actually ends: the database is a separate resource 

This is the caveat that matters most for a payment pipeline specifically, and it's the one "Kafka gives you exactly-once" glosses over. Kafka's transactional guarantee covers Kafka's own topics and consumer offsets. It does not extend to a database write in the same method, because the database and Kafka are two independent transactional resources with no shared coordinator. In the code above, ledgerService.postPending(event) runs in a normal JDBC or JPA transaction, entirely separate from the Kafka transaction wrapping the method. If the ledger write commits and the process crashes before the Kafka transaction commits, you have a ledger entry with no corresponding settled event. If it happens the other way — Kafka commits, the database write fails — you have a settled event with no ledger record behind it. 

Spring does offer a ChainedKafkaTransactionManager to synchronize a JDBC transaction manager with a Kafka one, but it's been deprecated since Spring Kafka 2.7, for a reason worth taking seriously rather than working around: chaining two independent transaction managers isn't real atomicity, it's sequencing — one commits, then the other does, and a crash in the gap between the two still leaves you inconsistent. True cross-resource atomicity would require distributed transactions (XA), which Kafka doesn't support natively and which carries enough of its own performance and complexity cost that it's rarely the right trade for a payment pipeline anyway. 

The pattern that actually holds up is the transactional outbox: write the ledger entry and a corresponding outbox row in the same database transaction — a single resource, genuinely atomic — then have a separate process (Debezium's CDC connector against the outbox table is the common choice) publish the outbox row to Kafka asynchronously, with its own retry and idempotent-publish logic. That process reintroduces at-least-once delivery for the outbound event, which is exactly why the idempotency-key pattern at the consuming service's API boundary still matters even after all of this is correctly wired — Kafka transactions eliminate duplicates within the pipeline; they don't remove the need for the receiving service to be able to safely process the same event twice if the outbox publisher retries. 

What this costs, and when it's worth paying 

Worth saying plainly for whoever's signing off on the architecture, not just the engineer implementing it: transactional writes carry real overhead — coordinator round-trips, read_committed consumers buffering until a transaction resolves, and Kafka 4.0's KRaft-mode improvements narrowing but not eliminating that gap. For a high-volume, latency-sensitive stream where a duplicate is genuinely harmless — a metrics pipeline, an audit log with its own downstream dedup — at-least-once with idempotent consumers is usually the better trade. For a payment-initiated-to-settled pipeline, where a duplicate is a double-charge, the latency cost of transactions is the cheaper problem to have. 

The shape of a pipeline that actually holds 

Put together, a payment pipeline that earns the phrase "exactly-once" end to end looks like this: idempotent, transactional producers with a stable transactional.id per instance; consumer offset commits folded into the same transaction via sendOffsetsToTransaction, which Spring wires up automatically behind @Transactional; every downstream consumer explicitly set to read_committed; the database write kept in its own atomic transaction alongside an outbox row rather than chained to Kafka's transaction manager; and an idempotency key at the API boundary of whatever service consumes the eventual outbound event, because the outbox publisher itself is back to at-least-once delivery by design. None of these pieces individually claims to solve the whole problem. Wired together deliberately, with the seams between them understood rather than assumed away, they do. 

Exactly-Once Semantics in Kafka Payment Pipelines 

A code-level walkthrough of Kafka transactions, producer idempotence, and consumer offset commits for payment event pipelines where "probably delivered once" isn't good enough.

Kafka guarantees exactly-once is true in a narrower sense than most payment teams assume. Here's what transactions cover, and where the gap still is

Transactional Producers and Consumers 

Blameless postmortem culture was built for a world where the only audience for an incident writeup was the team that lived through it. Write down what happened, name the contributing factors instead of the person, and let the retro run as long as it needs to before anyone commits to a root cause. That model works beautifully right up until a regulator's clock starts running the moment your team becomes aware of the incident — and a clock that starts at "aware," not "understood," doesn't wait for your retro to finish. This one's for the engineering team trying to keep both things true at once. 

Two clocks, two documents, two different jobs 

The tension isn't philosophical, it's structural. Blameless postmortem practice, in its SRE-textbook form, exists to protect a specific thing: an engineer's willingness to say "I did this" honestly, without editing the account to protect themselves, because the account is what the team learns from. That only works if nothing bad happens to the person who wrote it — no performance review consequence, no name attached to a document that leaves the team. 

Regulatory incident reporting has a different purpose. Under DORA, a major ICT-related incident triggers an initial notification as soon as possible, and in any case within four hours of classification as major and no later than 24 hours from the moment the financial entity became aware of the incident. That is followed by an intermediate report within 72 hours of the initial notification and a final report no later than one month after the intermediate report or latest updated intermediate report. NIS2's Article 23 establishes a similar reporting sequence for entities in its scope: a 24-hour early warning, a 72-hour incident notification, and a final report within one month. Neither framework is written for a team that has finished figuring out everything that happened. Both require information to be provided while the investigation may still be ongoing. 

That's the actual design problem: a postmortem culture built for one document, unlimited time, and no external reader now has to also produce a document with a fixed deadline, an external reader, and content that may exist long before the internal investigation is finished. Treating these as the same document, written once and reused, is where teams get hurt — either the regulatory filing goes out half-baked because the team was still protecting psychological safety by staying vague, or the blameless retro quietly stops being blameless because everyone knows whatever gets said might end up quoted in a document a supervisor reads. 

The teams that handle this well don't try to write one document that satisfies both purposes. They build two artifacts, on two tracks, governed by two different rules — and they make that separation explicit in the incident process itself, not something people improvise under pressure at hour three of an outage. 

The regulatory filing is fact-and-timeline, not narrative-and-cause. What DORA requires in the initial stages is specific, supportable information: when the incident was detected, what happened, which services and Member States were affected, how the incident was discovered, what containment or business continuity measures have been taken, and, where available, information about its origin. Root-cause information belongs in the final report, once there is evidence to support it. A provisional classification, clearly marked as provisional, is entirely different from inventing a confident root-cause narrative under four-hour pressure. The acceptance criterion worth writing into the incident process explicitly: while a major incident is classified as active, any regulatory filing SHALL state the incident's known facts and current status, and SHALL NOT assert a root cause beyond what current evidence directly supports. 

The blameless postmortem is cause-and-learning, not just fact-and-timeline. This is the document that takes the time it needs, involves the people who need to speak candidly, and stays internal by design — its value depends on that. It's where "the deploy pipeline let a config change skip review because the emergency-bypass flag doesn't expire" gets written down honestly, in a way nobody would write if they thought a regulator might read it verbatim next week. 

The two documents can, and should, differ in emphasis without disagreeing in fact. The regulatory filing says "a configuration change caused an authentication service outage affecting approximately 40,000 customers for 90 minutes." The postmortem says the same thing, plus the actual chain of contributing factors, plus what's changing about the deploy pipeline as a result. Neither document lies to the other's reader. They just answer different questions, for different reasons, on different clocks. 

Who signs the filing matters more than who wrote the retro 

Blameless culture protects the engineer who describes what happened. It was never meant to protect the organization from having to establish accountability for the regulatory filing itself — those are different roles, and conflating them is where a lot of the psychological-safety erosion actually starts. A named incident commander — a role, not necessarily always the same person — owns or coordinates the classification decision and filing process, is accountable for the submission being accurate and on time, and is explicitly not the same thing as "the person whose commit caused the outage." Separating those roles is what lets the engineer who wrote the honest account in the postmortem stay unnamed in it, while someone with actual authority stands behind what goes to the regulator. 

This distinction also keeps another consideration visible: personal data. Naming an individual engineer, their specific mistake, and their identity in a document that may be shared with a regulator or auditor raises data-protection considerations under GDPR that are separate from DORA or NIS2. The answer isn't necessarily to remove every reference to individuals, but to make sure the regulatory filing contains only the personal data that is actually necessary for its purpose. A filing built primarily around affected systems, facts and process gaps rather than unnecessary identification of individual engineers is more consistent with both data minimisation and the principles behind a blameless culture. 

One nuance worth building the process around explicitly 

For a financial entity in scope of both frameworks, DORA operates as the sector-specific lex specialis for the requirements it covers, meaning the DORA requirements apply instead of the corresponding NIS2 requirements in those areas. That does not mean NIS2 simply disappears from the picture: certain NIS2 provisions can continue to apply, and NIS2 remains directly relevant to entities or activities that fall within its scope outside the requirements displaced by DORA. The practical implication for a spec: don't hard-code "always file under both frameworks" into the incident process. Build the classification step to determine which regime and obligations actually apply to the entity and incident, and let the notification workflow branch from there — the same discipline we've written about before for any requirement that depends on a jurisdictional or regulatory fact that isn't fixed at design time. 

What actually survives contact with a real incident 

None of this works as a policy document nobody's read before 2 a.m. on the night it matters. The classification authority, the filing template, the separation between what goes to the regulator and what goes in the retro — all of it has to be decided and rehearsed before the clock starts, because a four-hour deadline leaves no room to invent a process while also running one. 

Blameless culture and regulatory notification aren't actually in conflict. They just can't be the same document, written for the same purpose, on the same clock — and the moment a team tries to make them one thing is the moment one of the two purposes quietly loses. 

Postmortems Without Blame  

How SRE-style postmortem practices adapt when incidents also trigger regulatory notification obligations.

Blameless postmortems and regulatory reporting want different things from one incident. Here's a process where DORA and NIS2 clocks don't erode either.

Building an Incident Response Culture in a Regulated Engineering Team 

"Just add an idempotency key" is the kind of advice that sounds complete and isn't. I've reviewed enough payment integrations to know the gap is never the header — every client remembers to send Idempotency-Key: <uuid>. The gap is what the server does with it once two requests carrying the same key arrive close enough together that "check if we've seen this before" and "record that we're processing it" aren't actually one atomic step. This one's for the backend engineer who's about to build that step and wants to see where it actually breaks. 

A client calls POST /payments, the request reaches your service, the charge succeeds, and the response times out on the way back. The client — correctly, per its own retry policy — sends the same request again. Nothing about HTTP tells your service these are "the same" request. As far as your payment endpoint is concerned, it received two POST calls, and unless something explicitly ties them together, it will process two charges. 

An idempotency key is that tie. The client generates a key — a UUIDv4 is the standard choice — and sends it once, reusing the identical value on every retry of that same logical operation. The server's job is to guarantee that no matter how many times a given key arrives, the underlying charge happens at most once, and every caller gets the same response. That guarantee sounds simple. Implementing it correctly requires answering three questions your design either handles explicitly or gets wrong silently: 

Same key, same payload, arriving again → return the original result, don't reprocess. 

Same key, different payload → reject. A key isn't a request cache; reusing it for a different amount or recipient is a client bug, and the server needs to catch it rather than silently process whichever payload arrived first. 

Same key, arriving concurrently — two requests racing each other before either has finished — → exactly one of them executes the charge; the other waits or is told to retry. 

Case 3 is the one most naive implementations miss, because it's the one that doesn't show up in a single-threaded test. It's also the one that actually causes double-charges in production, because it's exactly what a client's retry-on-timeout logic produces: the original request is still in flight, slow, and the client — having given up waiting — fires the retry while the first call is still running. 

The fast path: claiming a key atomically in Redis 

Redis is a good fit for the hot path because the operation you need — "check if this key exists, and if not, claim it, in one atomic step" — is exactly what SET key value NX gives you. NX means "only set if the key doesn't already exist," and Redis executes it as a single atomic command, which is what closes the race in case 3. 

    private final StringRedisTemplate redis; 

    private static final Duration LOCK_TTL = Duration.ofSeconds(30); 

    private static final Duration RESULT_TTL = Duration.ofDays(7); 

    public IdempotencyResult claim(String idempotencyKey, String payloadHash) { 

        String lockKey = "idem:lock:" + idempotencyKey; 

        String resultKey = "idem:result:" + idempotencyKey; 

        // Already completed? Return the stored response — this is the 

        // "same key, same payload, return original result" path. 

        String existing = redis.opsForValue().get(resultKey); 

            IdempotencyRecord record = deserialize(existing); 

            if (!record.payloadHash().equals(payloadHash)) { 

                throw new IdempotencyConflictException(idempotencyKey); 

            return IdempotencyResult.replay(record.response()); 

        // Not completed yet — try to claim the in-flight lock atomically. 

        Boolean acquired = redis.opsForValue() 

            .setIfAbsent(lockKey, payloadHash, LOCK_TTL); 

        if (Boolean.TRUE.equals(acquired)) { 

            return IdempotencyResult.proceed(); // caller does the real work 

        // Someone else is already processing this key right now. 

        String inFlightPayload = redis.opsForValue().get(lockKey); 

        if (inFlightPayload != null && !inFlightPayload.equals(payloadHash)) { 

            throw new IdempotencyConflictException(idempotencyKey); 

        throw new IdempotencyInProgressException(idempotencyKey); // 409, client retries later 

    public void recordResult(String idempotencyKey, String payloadHash, String response) { 

        redis.opsForValue().set(resultKey, serialize(payloadHash, response), RESULT_TTL); 

        redis.delete("idem:lock:" + idempotencyKey); 

 The payloadHash — a hash of the normalized request body, not the raw JSON — is what implements case 2. Two requests with the same key but different amounts produce different hashes, and the mismatch is caught explicitly instead of silently resolved in favor of whichever request happened to win the race. 

The controller wraps the actual charge logic around this: 

public ResponseEntity<PaymentResponse> createPayment( 

        @RequestHeader("Idempotency-Key") String idempotencyKey, 

        @RequestBody PaymentRequest request) { 

    String payloadHash = hashService.hash(request); 

    IdempotencyResult claim = idempotencyService.claim(idempotencyKey, payloadHash); 

        return ResponseEntity.ok(claim.response()); 

    PaymentResponse response = paymentService.charge(request); // the real work 

    idempotencyService.recordResult(idempotencyKey, payloadHash, serialize(response)); 

Redis gives you a fast, correctly-atomic claim. It doesn't give you durability guarantees a payment system should actually depend on. An eviction under memory pressure, a Redis failover that loses a few seconds of writes, or a TTL that expires while a downstream system is still retrying, and the in-memory record of "we already processed this" is gone — which reopens exactly the race the whole mechanism exists to close. 

The fix is the same one we've written about for reconciliation: don't trust a single system of record for something this consequential. The database — not Redis — should be the actual source of truth, enforced with a unique constraint: 

       uniqueConstraints = @UniqueConstraint(columnNames = "idempotency_key")) 

    @Column(name = "idempotency_key", nullable = false) 

Idempotency Keys in Practice  

An idempotency key sounds like the whole fix for duplicate payments. Here's where the guarantee breaks, and the Redis-plus-database pattern that closes it.

Preventing Duplicate Payments in Distributed Systems 

Most of what I test never gets talked about outside engineering, and reconciliation is the clearest example. Nobody in a board meeting asks how the nightly matching job performed. But when it quietly falls behind, or starts flagging false breaks nobody trusts, that's the moment a risk officer finds out about it — usually not on a convenient day. This one's for the people who own that risk, not the people who write the SQL. 

The job nobody notices, until it's the job that's late 

Every bank runs some version of the same nightly ritual: internal ledger entries get matched against an external settlement file — a card network, a correspondent bank, a payment processor — to confirm that what your system thinks happened actually happened. At low volume, this is unglamorous and forgettable. At the volumes we test for our banking clients — tens of millions of transactions a night, across multiple settlement partners with their own formats and their own timing — it stops being a background job and starts being infrastructure that either holds or doesn't. 

When it doesn't hold, the failure is rarely dramatic. It's a job that used to finish in ninety minutes now finishing in six hours, quietly eating into the operational window before the business day starts. It's a database that locks up under the load of a full-table comparison, degrading every other system sharing that instance overnight. It's a growing backlog of "unmatched" transactions that nobody has time to investigate individually, so they get waved through with a shrug — which is precisely the kind of control gap a regulator, an auditor, or DORA's operational resilience expectations will eventually ask you to explain. 

Why this is a risk conversation, not just an engineering one 

Reconciliation isn't a nice-to-have batch job. It's one of the primary controls a bank has for catching the things that matter most: a settlement that never arrived, a duplicate charge, a fee applied twice, a transaction posted against the wrong account. A reconciliation process that can't keep pace with transaction growth doesn't fail loudly — it fails by getting slower, then by getting less precise, then by training the operations team to trust it less, until someone stops actually reading the exception report and starts just clearing it. 

That's the gap I spend most of my time as a tester trying to close: proving, before a system goes live, that it will hold at the volumes it's actually going to see — not the volumes it was demoed at. A reconciliation engine that performs well on a sample dataset in a sprint demo and buckles at three years of real transaction growth is not a hypothetical failure mode. It's the single most common gap we find when we test these systems properly. 

What "engineered well" actually looks like 

The technical choices behind a reconciliation job that scales aren't exotic, but getting them right is the difference between a process that degrades gracefully as volume grows and one that hits a wall. 

Batching, not brute force. A naive reconciliation job pulls the full day's ledger and the full settlement file into memory and compares them wholesale. That works fine in a proof of concept and falls over the first time volume triples. The systems that hold up process in bounded batches — by account range, by settlement partner, by time window — so that a spike in one segment doesn't stall the entire nightly run, and so a failure in one batch can be retried without re-running everything else. 

Indexing for the comparison you're actually doing, not the one you started with. Reconciliation is fundamentally a matching problem, and matching problems live or die on whether the database can find candidate matches without scanning everything. A composite index aligned to how records are actually matched — transaction reference, amount, and settlement date together, rather than three separate single-column indexes — is usually the difference between a job that scales linearly with volume and one that scales quadratically and quietly stops finishing on time. 

Diffing incrementally, not from scratch every night. The most resilient designs we test don't re-diff the entire ledger against the entire settlement file every night. They diff what changed since the last confirmed match, carry forward previously reconciled records without re-touching them, and isolate the comparison to genuinely new or previously-unmatched activity. That single design decision is usually what separates a reconciliation job with a stable, predictable runtime from one whose runtime grows in lockstep with the size of the ledger itself — which, for a growing bank, means a job that gets slower every single quarter by design, not by accident. 

Where testing actually catches this before it becomes a production incident 

This is the part that doesn't show up in an architecture diagram: none of the choices above are safe to assume just because they're in the design document. We test reconciliation systems the way we'd test any other control that a regulator will eventually ask about — not just "does it match correctly on a clean dataset," but "does it still match correctly, and finish on time, at realistic volume, with realistic messiness." 

That means load-testing against a dataset sized for where the business will be in two years, not where it is today. It means deliberately injecting the ugly cases — a settlement file that arrives late, a duplicate record, a currency mismatch, a transaction that legitimately has no counterpart yet because it's still in flight — and confirming the system flags them correctly instead of either silently dropping them or burying the operations team in false positives. A reconciliation engine that's too noisy is nearly as dangerous as one that's too quiet: both end with a human learning to stop trusting the exception report, which is the actual control failure, whatever the underlying code looks like. 

It also means testing what happens when the job itself fails partway through — because at scale, it eventually will. Does a failed batch retry cleanly, or does it leave the ledger in a state where some records are double-counted? Does the system tell someone the job didn't finish, or does the absence of an alert just look like a quiet, uneventful night? 

A reconciliation process is one of the places where "it works" and "it's a control you can rely on under scrutiny" are genuinely different bars, and the gap between them only shows up at the volumes and the failure conditions a demo never covers. Getting it right isn't about writing more code — it's about designing the batching, indexing, and diffing strategy for the business's actual growth curve, and then proving it holds under the same pressure a live incident, an audit, or a regulator's operational resilience test eventually will. 

That's the standard we hold reconciliation systems to before we sign off on them, and it's the same standard we'd want any delivery partner testing a control this important to be held to. 

Nightly Reconciliation at Scale  

A reconciliation job that can't scale doesn't fail loudly — it gets slower, less precise, less trusted. Here's what testing it properly actually requires.

Matching Millions of Transactions Without Melting Your Database

Most eIDAS 2.0 content is written for compliance and legal teams, and it reads that way — obligations, deadlines, “member states shall.” None of it tells you what to actually put in a specification when the development team asks what the signup flow does on the Tuesday morning eIDAS 2.0 goes live in one country and stays exactly as it was in six others. This one’s for the BA holding that spec. 

The honest version of “engineer eIDAS 2.0 into the signup flow” isn’t a replacement project. It’s an addition to a system that already has to keep working, for years, across markets that are adopting the EU Digital Identity Wallet on completely different timelines. Acceptance criteria that only account for the wallet, without accounting for everywhere it isn’t live yet, are the ones that come back from engineering with the same question every time: “what happens for everyone else?” 

The rollout you’re specifying against is not uniform, and won’t be for years 

Under eIDAS 2.0 (Regulation (EU) 2024/1183), member states are required to offer at least one certified EUDI Wallet to citizens and businesses within 24 months of the relevant implementing acts being adopted — a timeline broadly expected to land in late 2026. As of Q2 2026, however, implementation across the EU-27 sits in three visibly different tiers. France, Italy, and Poland already have production or near-production wallets; Germany and Sweden are running beta programmes; a handful of smaller states haven’t published a national strategy at all. A homogeneous EU-wide launch on the current deadline is already considered unlikely by people tracking it closely — what arrives is a first wave of compliant-or-near-compliant launches with genuinely different functional scope market to market. And the deadline that actually matters for a bank’s obligations is later still: large platforms and regulated-sector organisations, banking included, are expected to be required to accept the wallet as a primary authentication method by late 2027. 

That gap — wallets appearing country by country from 2026, banks obligated to accept them EU-wide only from 2027 — is the entire shape of the specification problem. A signup flow spec written as “if wallet available, use wallet; else, fall back to eKYC” sounds complete, but it isn’t. “Available” isn’t a global boolean — it’s a per-country, per-issuer, slowly-shifting fact your system needs to be able to check and act on independently for each market it operates in, for a multi-year window, not a cutover weekend. That’s a feature-flagged capability matrix, not a runtime fallback — and it needs to be specified as one from the first version, because retrofitting per-country branching into a spec that assumed a single global switch is exactly the kind of rework a precise spec exists to avoid. 

The rail under the most pressure isn’t the one everyone’s planning around 

It’s tempting to treat the current eKYC rail as the stable baseline while the wallet integration is the hard new work. In this context, eKYC is the umbrella term for the set of remote identity-verification methods a bank already operates — and the specific rails within it deserve individual scrutiny. Germany’s own supervisory picture illustrates why. BaFin recognises four remote identification rails under the GwG: video identification, chip-based eID, qualified-electronic-signature-bundled flows, and automated document-plus-biometric verification. 

Video-ident — trained agents verifying a customer against ID document security features over a live video call — remains widely used, and it’s also the rail under the most active pressure right now. Deepfake and injection attacks target exactly that channel: a live video session is precisely the surface where synthetic media threats are most effective. 

That’s worth putting in front of engineering leadership as its own priority, independent of the eIDAS 2.0 timeline: if video-ident is the primary rail your onboarding flow depends on today, the acceptance criteria for liveness and injection-attack detection on that existing flow need attention now, not as a “we’ll harden it later, once the wallet takes over” deferral. The wallet migration is a multi-year project. The video-ident attack surface is a today problem wearing a future-looking regulation as an excuse to wait. 

What the wallet flow actually requires, in terms a developer can build against 

When a market’s wallet is live and a bank has completed relying-party registration, the technical shape of a presentation request is specific enough to write real acceptance criteria against, not just “integrate with the EUDI Wallet”: 

The signup flow SHALL request only the PID attributes required for the specific product being opened (name, date of birth, and residential address for a current account — not the full Person Identification Data set) via an OpenID4VP presentation request, honouring the wallet’s selective disclosure mechanism. 

WHEN a wallet returns a presented credential, the system SHALL verify the credential’s signature chain against the relevant Member State’s published trusted-issuer list before accepting any attribute value from it. 

WHEN a presented PID attestation does not carry “high” level of assurance as defined under eIDAS 2.0’s implementing regulation, the system SHALL reject wallet-based onboarding for that session and route the customer to the fallback eKYC rail — a “substantial” assurance attestation is not sufficient for account opening. 

WHEN a presented credential’s issuer certificate has been revoked, the system SHALL treat the presentation as failed and log the revocation check result, not merely the outcome. 

Two details matter enough to call out explicitly in a spec, because they’re the ones a developer building against a vaguer requirement will guess wrong. 

First, selective disclosure isn’t an implementation nicety — it’s the point of the credential format. A spec that asks the wallet integration to pull the full PID every time, “just in case a later screen needs it,” works against the framework’s actual design and creates a data-minimisation problem worth flagging to whoever owns privacy sign-off. 

Second, “high” assurance is not a formality to check once and forget. eIDAS 2.0’s implementing regulation ties this requirement to the ETSI TS 119 461 technical standard specifically, and a “substantial”-level attestation, which some issuance paths can produce, does not carry the same weight for AML customer due diligence. A spec that doesn’t distinguish assurance levels in its acceptance criteria is a spec that will let engineering treat any successful wallet presentation as equally trustworthy — which it isn’t. 

The dependency that isn’t a sprint — relying party registration 

A bank cannot start accepting wallet presentations by shipping code. Before any of the flow above is technically possible, the institution has to register its intent to rely on the EUDI Wallet with the relevant Member State authority — a formal, non-engineering process with its own lead time, run by legal and compliance, not by the delivery team. This belongs in a BA’s delivery plan as an explicit blocking dependency with its own timeline, tracked the same way an external vendor contract would be, not folded into “integration work” as if it were a configuration step the engineering team controls. A spec that’s otherwise perfect and ships against an assumption that registration is already complete is a spec that’s going to sit blocked on a legal process nobody scheduled for. 

Spec around the part that’s still genuinely unsettled, rather than waiting for it to resolve 

Here’s the honest gap: AMLR sets out what banks must achieve — reliable, risk-based customer due diligence — while eIDAS 2.0 and its technical standards describe how a high-assurance identity proofing process can look. What the industry has been waiting on is the granular Regulatory Technical Standards that say explicitly whether a high-assurance wallet presentation, on its own, fully satisfies a bank’s CDD obligation, or whether it’s one strong input among several a risk-based decision still has to weigh. 

That guidance has been described as still catching up even as wallets start going live — which means a spec written today can’t treat “the RTS said X” as a fixed input. It can only design for the range of reasonable answers. 

The move that survives whichever way that guidance lands is the same discipline we’ve written about before for specs that have to survive being read by something — or someone — other than the person who wrote them: treat wallet presentation as one signal into a risk-based decision engine that also runs sanctions screening, PEP checks, and whatever risk-scoring already gates a new account, rather than hard-coding “valid high-assurance presentation received” as an automatic full-CDD pass. If the eventual RTS says a high-assurance presentation is sufficient on its own, that’s a configuration change to how much weight the signal carries — not a rebuild of the decision engine. If it says otherwise, the architecture already assumed as much. Specifying flexibility into the exact point that’s still unresolved is cheaper, by a wide margin, than specifying certainty and being wrong about which certainty to specify. 

None of this is really a story about a new regulation. It’s the same discipline regulated banking delivery has always demanded of a requirements document — precise enough that an engineer building from it doesn’t have to guess, and honest enough about what’s still unresolved that the system built from it doesn’t have to be rebuilt when the answer arrives. eIDAS 2.0 just made sure that discipline has to hold across country borders and a rollout timeline nobody fully controls, instead of inside one team’s release calendar. 

Digital Identity Verification at Onboarding 

Most eIDAS 2.0 content is written for compliance teams. This is for the BA who has to turn "accept the EUDI Wallet" into acceptance criteria — while eKYC keeps running everywhere it isn't live yet.

eIDAS 2.0, content, compliance teams, BA, accept the EUDI Wallet, into acceptance criteria, while eKYC keeps running everywhere it isn't live yet.

Engineering eIDAS 2.0 and eKYC Into the Signup Flow 

Every core banking system you've worked on had exactly one client that mattered: your own front end, built by people who sat two desks away and could be pulled into a Slack thread when something looked wrong. Banking-as-a-Service inverts that. Now the client is a fintech program manager you've never met, integrating against your API on their own timeline, and the thing standing between their bug and your bank's balance sheet is whatever isolation you actually built — not whatever you assumed. This one's for the backend engineer being asked to open the core up. 

Embedded finance sounds like a product decision — a bank licenses its charter and infrastructure so a non-bank brand can offer cards, accounts, or payments under its own name. What it actually is, underneath the partnership deck, is a systems design problem: your ledger, your KYC pipeline, and your reconciliation process all need to serve multiple tenants who don't work for you, don't share your incident channel, and in some architectures don't even have direct visibility into whether their own numbers are correct. Get that architecture right and BaaS is a genuinely good distribution model. Get it wrong and you've built the exact shape of failure that took down one of the industry's largest middleware providers in 2024. 

The failure mode is a ledger problem before it's ever an API problem 

Synapse's collapse is worth understanding in technical terms, not just as a cautionary headline, because the root cause was architectural. Synapse operated as middleware between fintech apps and several partner banks, using "for benefit of" (FBO) omnibus accounts — one pooled account at the bank, holding funds for potentially thousands of end users, with Synapse's own systems tracking who owned what share of the pool. That per-user detail is what's sometimes called a shadow ledger: a secondary set of balances, held outside the bank's core, that the bank itself never directly reconciled against on any meaningful cadence. When Synapse failed, its partner banks lost access to Synapse's records and, with them, the only detailed account of which end user owned which slice of the pooled funds. The result was a reported $65–95 million shortfall between what the banks held and what was actually owed to end users, money that wasn't provably missing so much as unprovable, because no single system of record had ever been required to agree with the others in real time. 

The lesson isn't "don't use middleware". It's that a sub-ledger only deserves the name if it can reconcile back to the core ledger on demand. One assessment of multi-product ledger design puts it bluntly: a sub-ledger that can't reconcile back to the core on demand isn't a sub-ledger, it's a second source of truth, and a second source of truth is exactly what a regulated bank can't afford to discover it has, especially under stress, especially during a partner's bankruptcy, when "on demand" becomes "immediately, under a court's questioning." Any BaaS architecture where the bank's core isn't verifiably the reconciled system of record for every tenant's end-user balance is running the same risk Synapse's partner banks were running, just at whatever scale hasn't failed yet. 

Regulators have already drawn this conclusion. The OCC, Federal Reserve, and FDIC jointly issued a request for information in mid-2024 on bank-fintech arrangements specifically, following a wave of enforcement actions against banks whose fintech partnerships had outrun their risk management — the direction of travel is toward expecting banks to maintain their own verifiable records and direct oversight of end-user funds, not to rely on a partner's assurances that its books are correct. Building the ledger architecture to satisfy that expectation isn't a compliance checkbox layered on afterward. It's the schema design decision you make in week one. 

Tenant isolation: pick the failure mode you can live with 

Multi-tenant architecture has three broad shapes, and the tradeoff between them is really a tradeoff in blast radius. A standalone, single-tenant deployment per program gives the strongest isolation — one tenant's bug, load spike, or breach genuinely cannot touch another's data, because there's no shared infrastructure to leak across — at the highest operational and infrastructure cost. A database-per-tenant model keeps a single application layer but gives each tenant its own database, which is the pattern most SaaS platforms converge on as the practical balance: strong isolation, and a single-tenant restore or migration that can't accidentally touch anyone else's data. A pooled, shared-schema model with a tenant-discriminator column scales to the most tenants most cheaply, and it's also the pattern with the lowest isolation — a missing WHERE tenant_id = ? clause, a bug in the row-level filter, or a noisy tenant's query load is now everyone's problem. 

For anything touching customer funds — which, in a BaaS platform, is the entire point — the pooled discriminator model shouldn't be the default, and it's worth being explicit about why: the cost of the isolation failure is categorically different from a typical SaaS data leak. A cross-tenant bug in a project management tool shows the wrong task list to the wrong customer. A cross-tenant bug in a ledger shows the wrong balance, or worse, lets one program's transaction post against another program's funds. Schema-per-tenant or database-per-tenant should be the floor for ledger and account data specifically, even if lighter-weight shared infrastructure is acceptable for genuinely tenant-agnostic services like a public rates API. 

Hibernate ships native support for exactly this decision, through its MultiTenancyStrategy options — SCHEMA and DATABASE give you a real, separately-connected schema or database per tenant, resolved per request through a CurrentTenantIdentifierResolver; DISCRIMINATOR is the shared-schema, tenant-ID-column approach, and it's the one that depends entirely on every query, every JPA repository method, and every raw SQL statement correctly applying the tenant filter with no exceptions, forever, across every engineer who ever touches the codebase. That's not a criticism of the discriminator pattern in general — it's a real, legitimate architecture for the right workload — but for a ledger table, the failure mode of "someone shipped a query without the filter" is the kind of bug you want architecturally impossible, not merely code-reviewed against: 

public class TenantIdentifierResolver implements CurrentTenantIdentifierResolver<String> { 

    public String resolveCurrentTenantIdentifier() { 

        String tenantId = TenantContext.getCurrentTenant(); 

            throw new IllegalStateException( 

                "No tenant context set — refusing to resolve a default schema"); 

The line that matters most in that snippet isn't the resolver logic — it's the refusal to fall back to a default. A multi-tenant system that silently defaults to some tenant when the context is missing is a system that will eventually post one program's transaction into another program's schema during exactly the kind of request-scoped bug that's otherwise harmless in a single-tenant app. 

API design: the confused-deputy problem, again, with real money attached 

A BaaS API's tenants aren't just data boundaries, they're credential boundaries, and the same failure pattern that shows up in tool-server design shows up here: a shared, broadly-scoped service credential used across every partner integration means a bug or a compromised partner integration can act with the platform's full privilege instead of that one tenant's. Every program-level API key needs to be scoped narrowly to that program's own accounts and actions, validated on every request against the tenant context resolved from the credential itself — not from a parameter the caller supplies and the server trusts. If a request can specify which tenant's ledger to post against, rather than that being derived entirely from the authenticated caller's identity, you've built a confused deputy into the API surface, and it's only a matter of time before a client's bug or a malicious actor discovers it. 

Idempotency deserves the same weight it gets in any payment API, for the same reason: a partner integration retrying a failed account-opening or transfer call after a timeout has to be safe to retry, which means every mutating endpoint needs to accept and honor a client-supplied idempotency key, returning the original result on a repeat rather than executing the operation twice. And because BaaS partners build their own products against your webhooks — a card authorization, a KYC status change, a settlement completing — those webhooks need per-tenant signing secrets and replay protection, exactly because a fintech partner's own security posture isn't something you control, and a leaked webhook secret for one tenant shouldn't let anyone forge events for another. 

Rate limiting is the last piece, and it's worth framing the same way a bulkhead is framed in any resilience design: without a per-tenant cap, one partner's traffic spike — a marketing campaign that goes better than they expected, or a bug in their retry logic — degrades the API for every other partner sharing the same infrastructure. Isolating rate limits per program, not just globally, is what keeps one tenant's bad day from becoming every tenant's incident. 

Reconciliation is the product, not the afterthought 

Everything above is what keeps the ledger correct in the steady state. What actually would have caught Synapse-shaped drift early is treating reconciliation between the core ledger and any tenant-facing balance as a monitored, alerting system in its own right — not a nightly batch job whose failures get triaged whenever someone notices the report didn't run. Daily reconciliation is the documented minimum in most guidance on this; event-driven reconciliation, checking that every tenant-facing balance change has a matching, correctly-attributed core ledger entry within minutes rather than hours, is what actually catches drift while it's still a rounding error instead of a court filing. A reconciliation break should page someone the same way a failed payment does, because functionally, it's an early warning for the exact failure that turned a single middleware provider's mistake into a multi-bank, tens-of-millions-of-dollars mess that regulators are still writing guidance in response to. 

We've built ledger and account infrastructure for banks long before "embedded finance" was the term for it, and the discipline underneath a good BaaS platform isn't new — it's the same core-banking rigor around reconciliation, audit trails, and system-of-record integrity that regulated banking has always required, applied now to an API surface with tenants you don't manage instead of a front end your own team built. The architecture question was never really "how do we expose our infrastructure to partners." It was "how do we expose our infrastructure to partners without ever losing the ability to prove, on demand, whose money is whose." 

Embedded Finance and Banking-as-a-Service

Banking-as-a-Service turns your ledger into a multi-tenant system with partners you don't control. Here's the tenant isolation, API design, and reconciliation architecture that keeps it from becoming the next Synapse. 

Banking-as-a-Service turns your ledger into a multi-tenant system; tenant isolation, API design, and reconciliation architecture. 

All News

Categories

All News

Categories

Do you want to read more?

Subscribe now for our bimonthly newsletter!