Resilience4j in Production
Circuit Breakers and Retry Storms Against Third-Party Payment Gateways
Every Resilience4j tutorial demos the same failure: the downstream returns a clean 500, the circuit trips, everyone claps. That's not the incident that pages you at 2am. The one that does is the payment gateway that doesn't go down — it goes slow, 8 seconds instead of 200 milliseconds, still technically returning 200s, right up until your own connection pool is empty and it's your service that's down, not theirs. This one's for the backend engineer who's configured @Retry and @CircuitBreaker and assumed that was the hard part done.
A payment gateway integration that only had to survive clean failures wouldn't need much resilience engineering — catch the exception, retry once, move on. What it actually has to survive is degradation: elevated latency, intermittent timeouts, a backend that's still accepting requests but taking four times as long to answer them. That's the case the annotation-level tutorials skip, and it's the case that turns a third party's bad afternoon into your own outage if the defaults are left as they are.
The retry storm is the part your own retries cause
Microsoft's Azure Architecture Center has a name for the specific failure this produces: the retry storm antipattern. The mechanism is simple and it's exactly what an unbounded or naive retry policy produces — a downstream service becomes slow or partially unavailable, clients retry, the flood of retries adds load to a service that was already struggling, recovery gets actively prevented rather than assisted, and the failure cascades outward instead of staying contained. The antipattern isn't retrying — it's retrying without a cap on attempts, without backoff, without jitter, and without anything watching the aggregate picture across all your callers at once.
For a payment gateway specifically, the cascade has a second stage that's easy to miss until it's happened once: your own thread pool and HTTP connection pool are shared resources. A gateway call that hangs for 8 seconds instead of failing fast doesn't just make that one request slow — it holds a connection and a thread for 8 seconds that an unrelated request, one that has nothing to do with payments, might also be waiting on. Enough slow gateway calls piling up and your service isn't degraded because the gateway is degraded. It's degraded because you ran out of threads, and the gateway's problem became every endpoint's problem. That's the actual shape of "cascading into a full outage" — not a single dramatic failure, but resource exhaustion accumulating quietly until something unrelated tips over.
A circuit breaker tuned for slow, not just down
The default Resilience4j circuit breaker configuration will catch a gateway that returns errors. It won't catch one that's just slow, unless you explicitly tell it to — and "slow but technically successful" is the more common failure mode against a real payment gateway than a clean outage. CircuitBreakerConfig exposes slowCallDurationThreshold and slowCallRateThreshold specifically for this: calls that take longer than the duration threshold count as slow calls, and if the proportion of slow calls crosses the rate threshold, the circuit opens — independently of whether those calls technically succeeded.
1 resilience4j.circuitbreaker:
2 instances:
3 paymentGateway:
4 slidingWindowType: TIME_BASED
5 slidingWindowSize: 60 # evaluate the last 60 seconds of calls
6 minimumNumberOfCalls: 20 # don't judge on a handful of requests
7 failureRateThreshold: 50 # open if ≥50% of calls fail outright
8 slowCallDurationThreshold: 2s # anything over 2s counts as "slow"
9 slowCallRateThreshold: 50 # open if ≥50% of calls are slow
10 waitDurationInOpenState: 30s
11 permittedNumberOfCallsInHalfOpenState: 5 TIME_BASED over COUNT_BASED matters here specifically because gateway call volume isn't constant — a count-based window sized for peak traffic will take far too long to react during a quiet period, and one sized for quiet periods will trip on normal peak-hour variance. A time-based window judges the gateway on what actually happened in the last 60 seconds, regardless of how many calls that was. The 2-second slow-call threshold isn't arbitrary either — it should be your own p99 latency budget for a checkout flow, not the gateway's advertised SLA; the circuit breaker's job is protecting your service's user experience, not grading the vendor.
A retry policy that doesn't add fuel to the fire
The instinct to add maxAttempts: 5 because "the gateway is sometimes flaky" is exactly how a retry storm gets built one config value at a time. Cap attempts low — two or three, not five — and never retry without backoff:
1 resilience4j.retry:
2 instances:
3 paymentGateway:
4 maxAttempts: 3
5 waitDuration: 500ms
6 enableExponentialBackoff: true
7 exponentialBackoffMultiplier: 2
8 retryExceptions:
9 - java.net.SocketTimeoutException
10 - org.springframework.web.client.HttpServerErrorException
11 ignoreExceptions:
12 - com.oceanobe.payments.exceptions.DeclinedTransactionException ignoreExceptions deserves as much attention as retryExceptions for a payment integration specifically — a card decline or an invalid-request error is not a transient failure, and retrying it doesn't just waste a call, it can retry a business decision that was already correct the first time. Only timeouts and 5xx-class failures belong in the retry path.
Exponential backoff configured through YAML alone still has a gap worth knowing about: it produces the same wait sequence for every instance of your service, which means if you're running a fleet of pods and the gateway degrades, every pod's retries land in near-synchronized waves — a smaller, self-inflicted version of the same thundering-herd problem the circuit breaker is there to prevent. Genuine jitter needs the programmatic config:
1 IntervalFunction backoffWithJitter = IntervalFunction.ofExponentialRandomBackoff(
2 500, // initial interval, ms
3 2.0, // multiplier
4 0.5 // randomization factor
5 );
6
7 RetryConfig retryConfig = RetryConfig.custom()
8 .maxAttempts(3)
9 .intervalFunction(backoffWithJitter)
10 .build(); The randomization factor is what breaks the synchronization across your own fleet — each pod's retries land at a slightly different offset instead of in lockstep, which is the difference between a brief load spike on the gateway and a self-inflicted thundering herd hitting it in unison.
The aspect-order trap that skews the circuit breaker's own math
Here's the one that catches teams who've configured everything above correctly and still see the circuit breaker behave strangely: Resilience4j's default aspect ordering places @Retry outside @CircuitBreaker. That's documented, reproducible behavior, not a hypothetical — Resilience4j's own default aspect order values put Retry at a lower precedence number than CircuitBreaker, which makes Retry the outer wrapper. The practical effect: when a call fails and gets retried three times, the circuit breaker sees three separate failures, not one logical operation that failed and was retried. A single degraded request gets recorded as three data points against your failureRateThreshold, silently skewing the exact statistic the circuit breaker's open/close decision depends on.
The fix is two explicit properties, flipping which aspect wraps which:
1 resilience4j.circuitbreaker:
2 circuitBreakerAspectOrder: 1
3 resilience4j.retry:
4 retryAspectOrder: 2 With CircuitBreaker as the outer aspect, it evaluates once per logical call — closed means the whole retry sequence is allowed to run and only the final outcome counts toward the failure rate; open means the retry sequence never starts at all, and none of those three attempts touch the network. That second part is the one that actually matters for a retry storm: get the order wrong, and a circuit that's already open still lets every retry attempt independently hit CallNotPermittedException inside the retry loop — cheap locally, but it means your retry policy and your circuit breaker aren't actually coordinated, they're just coincidentally stacked.
Bulkheads: bounding the blast radius when the circuit hasn't tripped yet
A circuit breaker protects you once it's decided the gateway is bad. Before that threshold is crossed — while calls are slow but not yet slow or frequent enough to open the circuit — a bulkhead is what stops those in-flight calls from consuming every thread your service has:
1 resilience4j.thread-pool-bulkhead:
2 instances:
3 paymentGateway:
4 maxThreadPoolSize: 10
5 coreThreadPoolSize: 5
6 queueCapacity: 20 A thread-pool bulkhead gives gateway calls their own isolated executor, capped independently of whatever pool serves the rest of your application — so ten hung gateway calls cost you ten threads from a dedicated pool of ten, not ten threads out of the pool your account-balance endpoint also depends on. That isolation is what actually delivers on "stop a slow gateway from cascading into a full outage": the circuit breaker decides when to stop calling the gateway at all, but the bulkhead is what keeps a gateway that's merely slow — not yet slow enough to trip the breaker — from taking the rest of your service down with it in the meantime.
What this looks like end to end, and why the wiring order is the whole point
Composed correctly — CircuitBreaker outermost, Retry inside it, Bulkhead governing the underlying thread pool the calls actually run on — the failure sequence during a real gateway degradation looks like this: calls start taking 3-4 seconds instead of 300ms; the bulkhead caps how many of those slow calls can be in flight at once, so the rest of your service keeps its own thread pool; the circuit breaker's sliding window accumulates slow-call and failure data across genuine logical operations, not inflated retry counts; once the slow-call rate crosses the threshold, the circuit opens and every subsequent call fails fast locally, with zero additional load reaching the gateway, for the entire waitDurationInOpenState window; after that window, a handful of half-open probe calls test whether the gateway has recovered before the circuit fully closes again. Compare that to the unconfigured default: retries stacking on top of retries across every pod, each one counted wrong by a breaker that isn't actually coordinated with them, threads exhausting from a shared pool, and a vendor's bad five minutes turning into your own outage that lasts considerably longer than five minutes.
One more thing worth building alongside all of this, specific to payments: none of this retry discipline is safe unless the calls being retried are actually idempotent. A payment gateway integration that retries a charge request without an idempotency key isn't protecting against failure — it's risking a duplicate transaction on top of it. Adyen's own API guidance is the pattern worth following regardless of which gateway you're integrating: a client-generated UUIDv4 idempotency key on every payment request, held valid for at least a week, so that a retried request after a timeout returns the original result instead of processing the charge a second time. Resilience4j controls when and how often you retry. It has no opinion on whether retrying is safe — that's a contract your integration has to establish with the gateway independently, and skipping it turns your resilience configuration into a mechanism for reliably duplicating charges instead of reliably avoiding downtime.
We've built enough payment integrations to have learned this the unglamorous way: the configuration that survives a real incident isn't the one that looks complete in a demo, it's the one where circuit breaker, retry, and bulkhead were wired with an explicit understanding of which one wraps which, and where "safe to retry" was answered before "how many times to retry" was.
