Chaos Engineering for Payment Systems

Under the Digital Operational Resilience Act, financial institutions no longer get to treat operational resilience as a documentation exercise. Companies are required to test ICT systems by DORA in realistic failure scenarios, and for institutions designated as significant, that includes threat-led penetration testing and advanced resilience testing under the Regulatory Technical Standards on testing. The institutions that will meet this requirement with confidence are the ones already treating deliberate failure injection — chaos engineering — as a standing engineering practice, not a project spun up in response to a supervisory letter. 

This matters most on payment rails specifically. A payments platform failure isn't an inconvenience measured in a support ticket queue; it's a regulatory incident, a customer trust event, and in the case of critical infrastructure, potentially a reportable outage under DORA's incident classification thresholds. Chaos Engineering closes shut the gap between "we believe our payment system is resilient" and "we have evidence our payment system is resilient" is exactly what chaos engineering closes. 

What DORA Actually Requires — And Why That's a Floor, Not a Ceiling 

DORA's operational resilience testing obligations center on ICT risk management: institutions must be able to demonstrate they've tested their systems' ability to withstand, respond to, and recover from ICT-related disruptions, with the depth of testing scaled to the criticality of the system. For systems supporting payment processing, that criticality classification typically sits at the top of the scale, meaning the most rigorous testing obligations apply. 

It's worth being precise about what this covers and what it doesn't. DORA is about ICT and operational resilience — availability, integrity, and recoverability of systems and data under disruption. It is not a model governance framework, and testing conducted for DORA purposes shouldn't be conflated with the EU AI Act's requirements around algorithmic explainability or the ECB's model risk guidance for institutions using internal models. Chaos engineering, correctly scoped, is squarely an ICT resilience practice: it answers "does our system survive this failure," not "can we explain this decision." 

The regulatory testing calendar — annual for most obligations, every three years for threat-led penetration testing at significant institutions — sets a floor, not a target. An institution that only exercises failure scenarios when a regulatory test window requires it is, by definition, discovering resilience gaps at the worst possible cadence: too infrequently to catch issues introduced by the dozens of production changes that happen between test cycles. 

From Compliance Exercise to Engineering Discipline 

Chaos engineering, as an ongoing practice embedded in how a payments platform is built and operated, changes the shape of this problem entirely. Instead of a point-in-time test designed to satisfy an auditor, failure injection becomes a continuous input into how the system is designed, deployed, and monitored. 

On a payment rail built around an event-driven architecture — Kafka for transaction event streaming, Flink for real-time processing such as fraud scoring or transaction enrichment — the failure modes worth deliberately exercising map closely to the dependencies the platform actually has: 

Broker and partition failure. 

What happens to in-flight payment events when a Kafka broker goes down mid-transaction, or a partition leader election is triggered under load? Does the consumer resume cleanly, or is there a window where events are lost or double-processed? 

Processing job degradation. 

If a Flink job responsible for real-time fraud scoring falls behind or crashes, does the payment pipeline fail safely — holding transactions for review — or does it silently bypass the check under backpressure? 

Downstream dependency failure. 

Payment rails depend on card scheme connectivity, core banking system availability, and often a third-party fraud or KYC provider. Simulating latency spikes or outright unavailability in each of these surfaces exactly where a payment gets stuck, times out, or — the failure mode regulators care about most — gets processed inconsistently across a partial outage. 

Data consistency under partial failure. 

For any system where a payment event might be partially processed — debited on one side, not yet credited on the other, because a downstream step failed — deliberately inducing that partial failure in a controlled environment is the only reliable way to confirm reconciliation and compensation logic actually works, rather than assuming it does because it's never been triggered. 

The output of exercises like these isn't just "the system survived" or "the system didn't." It's a concrete, evidenced list of resilience gaps — the ones that would otherwise surface for the first time during an actual production incident, or during a regulator-mandated test where discovering a gap is a finding, not a routine engineering task. 

Building the Practice, Not Just Running the Test 

The institutions that get the most value from this approach treat chaos engineering as infrastructure, not as a one-off event: 

Start in non-production, controlled environments with realistic traffic patterns and data volumes, before any failure injection touches systems processing real payments. The goal is building confidence and fixing what breaks, not proving a point under live risk. 

Automate failure injection into the deployment pipeline so that resilience testing happens on a cadence tied to change frequency, not to the regulatory calendar. A payment rail that changes weekly and is only chaos-tested annually has fifty-one weeks of unverified assumptions between tests. 

Instrument for observability before you inject failure. Chaos engineering without detailed tracing, metrics, and logging tells you a system failed, but not where or why — which defeats the purpose. The engineering investment in observability and the investment in chaos testing are inseparable in practice. 

Document findings and remediation as part of the resilience testing record. This is where the compliance and engineering value converge: a well-run chaos engineering practice generates exactly the kind of evidence — documented failure scenarios, observed system behavior, remediation actions taken — that DORA's testing obligations require institutions to be able to produce. 

The Business Case Beyond Compliance 

Framed purely as a regulatory obligation, chaos engineering looks like a cost center. Framed accurately, it's closer to insurance with a return: institutions that find their payment rail's failure modes in a controlled exercise find them without a customer-facing outage, without a regulatory incident report, and without the reputational cost that comes with a payment platform failing publicly. The cost of discovering a reconciliation gap during a scheduled chaos exercise is an engineering afternoon. The cost of discovering the same gap during an unplanned production incident is measured very differently — in customer trust, in regulatory scrutiny, and in the operational cost of an incident response under pressure rather than a planned test under control. 

There's also a governance dimension that's easy to underweight. When a chaos engineering practice is mature and continuous, the conversation with a regulator or an internal audit function shifts from "prove to us this will work" to "here is eighteen months of evidence that it does, and here is what we changed each time we found something." That difference matters in a supervisory review — it's the difference between an institution reconstructing resilience evidence under deadline pressure and one that can produce it as a matter of course, because producing it is simply part of how the platform is run. 

DORA sets the minimum bar for how often this testing has to happen and how it has to be documented. The institutions best positioned to clear that bar comfortably are the ones that were already doing this before the regulation required it — and for whom the next test window is a formality rather than a scramble.