The Engineering Pattern Every Bank Integration Needs
Idempotency in Payment APIs
Duplicate payments caused by retries, network timeouts, or client failures are one of the most common — and costly — issues in payment system integrations. This article covers idempotency key design, server-side deduplication logic, the difference between at-least-once and exactly-once delivery, and how to implement this correctly in REST and event-driven payment flows.
The Problem: Why Duplicate Payments Happen
Consider a seemingly simple payment flow: a client sends a POST /payments request to initiate a charge. The payment processor receives the request, charges the card, and starts composing the response. Then the network drops. The client never gets an HTTP 200. What does it do?
In most retry implementations — including those built on top of resilience libraries like Resilience4j or Spring Retry — the client retries the request. The payment processor receives it again, has no memory of the first attempt, and charges the card a second time.
This is not a hypothetical edge case. It is one of the most commonly reported production incidents in fintech engineering, and the root cause falls into a handful of categories:
Network timeouts — the response is lost in transit, but the server completed the operation
Client crashes — the application restarts mid-request and retries from scratch
Load balancer retries — reverse proxies silently retry 5xx responses to upstream services
Message broker redelivery — Kafka, RabbitMQ, and SQS all guarantee at-least-once delivery by default
Webhook retries — third-party payment providers retry webhook delivery until they receive a 2xx acknowledgment
Orchestration failures — saga coordinators, workflow engines (Temporal, AWS Step Functions), or manual compensation logic trigger duplicate commands after partial failures
The financial and reputational cost of duplicate charges is severe. Chargebacks, compliance violations, customer trust erosion, and manual remediation work can dwarf the engineering effort required to prevent them in the first place.
The solution is idempotency.
What Idempotency Actually Means
An operation is idempotent if applying it multiple times produces the same result as applying it once.
In HTTP terms, GET, PUT, and DELETE are defined as idempotent by the specification. POST is not — but in payment systems, we need idempotent POST operations. We achieve this by adding client-supplied idempotency keys and server-side deduplication logic.
The guarantee we want:
Given the same idempotency key, the server processes the payment exactly once and returns the same response on every subsequent call — whether or not the first attempt succeeded.
This is different from safety (which means no side effects) and different from retryability (which merely means the client can retry). True idempotency is about the server being responsible for deduplication, not just the client being careful.
Idempotency Key Design
What Makes a Good Idempotency Key?
An idempotency key must be:
Unique per logical operation — not per HTTP request, but per business intent (e.g., "charge customer X for order Y")
Deterministic — regeneratable from business context, so it survives client crashes and restarts
Scoped correctly — unique within an appropriate namespace (per merchant, per user, per integration)
Short-lived or long-lived intentionally — retention policy must match business requirements
Key Generation Strategies
Strategy 1: UUID v4 (simple, stateless)
String idempotencyKey = UUID.randomUUID().toString();
Simple and collision-resistant, but not deterministic. If the client crashes before recording the key, it cannot regenerate it and must issue a new one — which is usually the correct behavior for interactive flows.
Strategy 2: Deterministic UUID v5 (content-based)
import java.util.UUID;
public class IdempotencyKeyGenerator {
// A stable namespace UUID for your organization
private static final UUID NAMESPACE = UUID.fromString("6ba7b810-9dad-11d1-80b4-00c04fd430c8");
public static String generate(String merchantId, String orderId, String currency, long amountCents) {
String content = String.join(":", merchantId, orderId, currency, String.valueOf(amountCents));
return UUID.nameUUIDFromBytes(
(NAMESPACE + content).getBytes(StandardCharsets.UTF_8)
).toString();
}
}
This is ideal for background jobs, scheduled payments, and any flow where the client must regenerate the same key after a crash.
Strategy 3: Composite structured key
{tenant_id}:{resource_type}:{business_id}:{intent}:{version}
Example: merchant_42:payment:order_9871:charge:v1
Human-readable and debuggable, but must be hashed or normalized before storage to avoid injection risks and enforce length limits.
Key Scoping and Namespacing
Always scope idempotency keys to a security boundary. A key from merchant A must never conflict with or match a key from merchant B.
// Bad: global key lookup
repository.findByKey(idempotencyKey);
// Good: key scoped to the authenticated merchant
repository.findByMerchantIdAndKey(authenticatedMerchantId, idempotencyKey);
Key Retention and Expiry
Use Case
Recommended Retention
Interactive payment (card present)
24 hours
Asynchronous payment initiation
7 days
Scheduled/recurring payment
30 days
Refund or reversal
90 days
After expiry, a reused key should be treated as a new request. Clearly document this contract in your API specification.
At-Least-Once vs Exactly-Once Delivery
Understanding delivery semantics is foundational to designing idempotent systems.
At-Least-Once Delivery
The infrastructure guarantees the message or request will be delivered, but may deliver it more than once. This is the default for:
HTTP retries (any retry without deduplication)
Kafka consumers with manual offset commits
AWS SQS standard queues
Webhook delivery systems
Consequence: Your processing logic will see duplicates. Idempotency must be implemented at the application layer.
At-Most-Once Delivery
The infrastructure will deliver the message zero or one times. No duplicates, but possible data loss. Used in fire-and-forget scenarios (UDP, some logging pipelines). Never appropriate for financial transactions.
Exactly-Once Delivery
The holy grail: each message processed exactly once, with no duplicates and no loss.
True exactly-once delivery is provably impossible across distributed systems without coordination. What practitioners call "exactly-once" is more precisely effectively-once: idempotent consumers paired with at-least-once delivery infrastructure, achieving the observable effect of exactly-once processing.
Effectively-Once = At-Least-Once Delivery + Idempotent Consumer
This is the model you should design for in payment systems. The infrastructure retries; your application deduplicates.
Server-Side Deduplication Logic
The Core Algorithm
The server-side idempotency check follows a consistent pattern:
1. Extract idempotency key from request
2. Look up key in the idempotency store (scoped to tenant/merchant)
3. If found AND in terminal state → return stored response
4. If found AND in-progress state → return 409 Conflict or 202 Accepted
5. If not found → atomically claim the key, process the request, store the result
State Machine for Idempotency Records
┌─────────────────────────────────┐
New Request │ │
──────────► [PENDING] ──► [PROCESSING] ──► [COMPLETED]
│
└──► [FAILED]
State
Meaning
Client Response
PENDING
Key claimed, processing not yet started
202 Accepted
PROCESSING
Actively being processed
409 Conflict (retry later)
COMPLETED
Finished successfully
Stored 2xx response
FAILED
Terminal failure
Stored 4xx/5xx response
Atomic Key Claiming
The most critical operation is atomically claiming the key to prevent race conditions when concurrent retries arrive simultaneously.
Using a database unique constraint (recommended):
CREATE TABLE idempotency_records (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
merchant_id VARCHAR(64) NOT NULL,
idempotency_key VARCHAR(255) NOT NULL,
status VARCHAR(20) NOT NULL DEFAULT 'PENDING',
request_hash VARCHAR(64), -- SHA-256 of request body
response_status INTEGER,
response_body JSONB,
created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
updated_at TIMESTAMPTZ NOT NULL DEFAULT now(),
expires_at TIMESTAMPTZ NOT NULL,
UNIQUE (merchant_id, idempotency_key)
);
CREATE INDEX idx_idempotency_expires ON idempotency_records (expires_at);
The UNIQUE constraint on (merchant_id, idempotency_key) is your deduplication guarantee at the database level. An INSERT that violates this constraint will fail atomically — no business logic runs twice.
Implementing Idempotency in REST APIs with Java/Spring
Project Structure Overview
payment-service/
├── api/
│ ├── PaymentController.java
│ └── IdempotencyFilter.java (or interceptor)
├── domain/
│ ├── PaymentService.java
│ └── IdempotencyService.java
├── persistence/
│ ├── IdempotencyRecord.java
│ └── IdempotencyRepository.java
└── config/
└── IdempotencyConfig.java
Step 1: The Idempotency Record Entity
@Entity
@Table(name = "idempotency_records",
uniqueConstraints = @UniqueConstraint(columnNames = {"merchant_id", "idempotency_key"}))
public class IdempotencyRecord {
public enum Status {
PENDING, PROCESSING, COMPLETED, FAILED
}
@Id
@GeneratedValue
private UUID id;
@Column(name = "merchant_id", nullable = false)
private String merchantId;
@Column(name = "idempotency_key", nullable = false)
private String idempotencyKey;
@Enumerated(EnumType.STRING)
@Column(nullable = false)
private Status status = Status.PENDING;
@Column(name = "request_hash")
private String requestHash; // SHA-256 of the request body
@Column(name = "response_status")
private Integer responseStatus;
@Column(name = "response_body", columnDefinition = "jsonb")
private String responseBody;
@Column(name = "created_at", nullable = false, updatable = false)
private Instant createdAt = Instant.now();
@Column(name = "updated_at", nullable = false)
private Instant updatedAt = Instant.now();
@Column(name = "expires_at", nullable = false)
private Instant expiresAt;
// Getters, setters, constructors omitted for brevity
}
Step 2: The Idempotency Repository
@Repository
public interface IdempotencyRepository extends JpaRepository<IdempotencyRecord, UUID> {
Optional<IdempotencyRecord> findByMerchantIdAndIdempotencyKey(
String merchantId, String idempotencyKey
);
@Modifying
@Query("""
UPDATE IdempotencyRecord r
SET r.status = :status,
r.responseStatus = :responseStatus,
r.responseBody = :responseBody,
r.updatedAt = :now
WHERE r.merchantId = :merchantId
AND r.idempotencyKey = :key
""")
int finalizeRecord(
@Param("merchantId") String merchantId,
@Param("key") String key,
@Param("status") IdempotencyRecord.Status status,
@Param("responseStatus") int responseStatus,
@Param("responseBody") String responseBody,
@Param("now") Instant now
);
@Modifying
@Query("DELETE FROM IdempotencyRecord r WHERE r.expiresAt < :now")
int deleteExpiredRecords(@Param("now") Instant now);
}
Step 3: The Idempotency Service
@Service
@Slf4j
public class IdempotencyService {
private final IdempotencyRepository repository;
private final ObjectMapper objectMapper;
private static final Duration DEFAULT_TTL = Duration.ofDays(7);
public IdempotencyService(IdempotencyRepository repository, ObjectMapper objectMapper) {
this.repository = repository;
this.objectMapper = objectMapper;
}
/**
* Attempts to claim an idempotency key for a new request.
* Returns the existing record if the key was already seen.
*
* @return Either an existing IdempotencyRecord (duplicate) or an empty Optional (new request)
*/
@Transactional(isolation = Isolation.READ_COMMITTED)
public Optional<IdempotencyRecord> findOrCreate(
String merchantId,
String idempotencyKey,
String requestHash) {
// Fast path: key already exists
Optional<IdempotencyRecord> existing =
repository.findByMerchantIdAndIdempotencyKey(merchantId, idempotencyKey);
if (existing.isPresent()) {
IdempotencyRecord record = existing.get();
// Guard: same key, different request body = bad client behavior
if (record.getRequestHash() != null
&& !record.getRequestHash().equals(requestHash)) {
throw new IdempotencyConflictException(
"Idempotency key reused with a different request body. " +
"Key: " + idempotencyKey
);
}
return existing;
}
// Slow path: try to insert a new record
try {
IdempotencyRecord record = new IdempotencyRecord();
record.setMerchantId(merchantId);
record.setIdempotencyKey(idempotencyKey);
record.setRequestHash(requestHash);
record.setStatus(IdempotencyRecord.Status.PROCESSING);
record.setExpiresAt(Instant.now().plus(DEFAULT_TTL));
repository.saveAndFlush(record);
return Optional.empty(); // Signal: this is a new, unclaimed request
} catch (DataIntegrityViolationException e) {
// Race condition: another thread claimed the key first
log.info("Race condition on idempotency key [{}] for merchant [{}]. " +
"Returning existing record.", idempotencyKey, merchantId);
return repository.findByMerchantIdAndIdempotencyKey(merchantId, idempotencyKey);
}
}
@Transactional
public void markCompleted(String merchantId, String key, int httpStatus, Object responseBody) {
try {
String bodyJson = objectMapper.writeValueAsString(responseBody);
repository.finalizeRecord(
merchantId, key,
IdempotencyRecord.Status.COMPLETED,
httpStatus, bodyJson, Instant.now()
);
} catch (JsonProcessingException e) {
throw new RuntimeException("Failed to serialize response for idempotency record", e);
}
}
@Transactional
public void markFailed(String merchantId, String key, int httpStatus, Object errorBody) {
try {
String bodyJson = objectMapper.writeValueAsString(errorBody);
repository.finalizeRecord(
merchantId, key,
IdempotencyRecord.Status.FAILED,
httpStatus, bodyJson, Instant.now()
);
} catch (JsonProcessingException e) {
throw new RuntimeException("Failed to serialize error for idempotency record", e);
}
}
public static String hashRequest(String requestBody) {
try {
MessageDigest digest = MessageDigest.getInstance("SHA-256");
byte[] hash = digest.digest(requestBody.getBytes(StandardCharsets.UTF_8));
return HexFormat.of().formatHex(hash);
} catch (NoSuchAlgorithmException e) {
throw new RuntimeException("SHA-256 not available", e);
}
}
}
Step 4: The Payment Controller with Idempotency
@RestController
@RequestMapping("/v1/payments")
@Slf4j
public class PaymentController {
private final PaymentService paymentService;
private final IdempotencyService idempotencyService;
private final ObjectMapper objectMapper;
public PaymentController(PaymentService paymentService,
IdempotencyService idempotencyService,
ObjectMapper objectMapper) {
this.paymentService = paymentService;
this.idempotencyService = idempotencyService;
this.objectMapper = objectMapper;
}
@PostMapping
public ResponseEntity<PaymentResponse> createPayment(
@RequestHeader(value = "Idempotency-Key", required = true) String idempotencyKey,
@RequestBody @Valid PaymentRequest request,
@AuthenticationPrincipal MerchantPrincipal merchant) throws JsonProcessingException {
// 1. Validate idempotency key format
if (!isValidKey(idempotencyKey)) {
return ResponseEntity.badRequest()
.body(PaymentResponse.error("Invalid Idempotency-Key format"));
}
// 2. Hash the request body for conflict detection
String requestBody = objectMapper.writeValueAsString(request);
String requestHash = IdempotencyService.hashRequest(requestBody);
// 3. Find or claim the idempotency key
Optional<IdempotencyRecord> existingRecord;
try {
existingRecord = idempotencyService.findOrCreate(
merchant.getId(), idempotencyKey, requestHash
);
} catch (IdempotencyConflictException e) {
// Same key, different body = client bug
return ResponseEntity.unprocessableEntity()
.body(PaymentResponse.error(e.getMessage()));
}
// 4. Duplicate request: return the cached response
if (existingRecord.isPresent()) {
IdempotencyRecord record = existingRecord.get();
return handleExistingRecord(record);
}
// 5. New request: process the payment
try {
PaymentResponse response = paymentService.charge(request, merchant);
idempotencyService.markCompleted(merchant.getId(), idempotencyKey, 201, response);
return ResponseEntity.status(HttpStatus.CREATED)
.header("Idempotency-Key", idempotencyKey)
.header("Idempotency-Replayed", "false")
.body(response);
} catch (PaymentDeclinedException e) {
// Business failure — still idempotent, cache the failure
PaymentResponse errorResponse = PaymentResponse.declined(e.getMessage());
idempotencyService.markFailed(merchant.getId(), idempotencyKey, 422, errorResponse);
return ResponseEntity.unprocessableEntity().body(errorResponse);
} catch (Exception e) {
// Infrastructure failure — do NOT cache; allow retry
log.error("Infrastructure failure processing payment [idempotency-key={}]",
idempotencyKey, e);
// The record stays in PROCESSING state; client may retry
return ResponseEntity.status(HttpStatus.SERVICE_UNAVAILABLE)
.body(PaymentResponse.error("Temporary failure. Please retry."));
}
}
private ResponseEntity<PaymentResponse> handleExistingRecord(IdempotencyRecord record) {
return switch (record.getStatus()) {
case PROCESSING -> ResponseEntity.status(HttpStatus.CONFLICT)
.body(PaymentResponse.error("Request is already being processed. Please retry later."));
case COMPLETED, FAILED -> {
try {
PaymentResponse cached = objectMapper.readValue(
record.getResponseBody(), PaymentResponse.class
);
yield ResponseEntity.status(record.getResponseStatus())
.header("Idempotency-Key", record.getIdempotencyKey())
.header("Idempotency-Replayed", "true")
.body(cached);
} catch (JsonProcessingException e) {
log.error("Failed to deserialize cached idempotency response", e);
yield ResponseEntity.status(HttpStatus.INTERNAL_SERVER_ERROR)
.body(PaymentResponse.error("Internal error retrieving cached response."));
}
}
default -> ResponseEntity.status(HttpStatus.INTERNAL_SERVER_ERROR)
.body(PaymentResponse.error("Unknown idempotency record state."));
};
}
private boolean isValidKey(String key) {
// Enforce UUID format, or your custom key format
return key != null && key.length() >= 16 && key.length() <= 255
&& key.matches("[a-zA-Z0-9_\\-:.]+");
}
}
Step 5: Scheduled Cleanup Job
@Component
@Slf4j
public class IdempotencyCleanupJob {
private final IdempotencyRepository repository;
public IdempotencyCleanupJob(IdempotencyRepository repository) {
this.repository = repository;
}
@Scheduled(cron = "0 0 3 * * *") // 3 AM daily
@Transactional
public void purgeExpiredRecords() {
int deleted = repository.deleteExpiredRecords(Instant.now());
log.info("Purged {} expired idempotency records", deleted);
}
}
Idempotency in Event-Driven Payment Flows
REST APIs are only part of the picture. Modern payment systems are event-driven, and message brokers introduce their own duplication challenges.
The Kafka Consumer Pattern
When consuming payment events from Kafka, the consumer must implement idempotent processing. Kafka's consumer groups guarantee at-least-once delivery by default; duplicate messages are expected after rebalances or consumer crashes.
@Service
@Slf4j
public class PaymentEventConsumer {
private final IdempotencyService idempotencyService;
private final PaymentService paymentService;
private static final String SYSTEM_TENANT = "kafka-consumer";
public PaymentEventConsumer(IdempotencyService idempotencyService,
PaymentService paymentService) {
this.idempotencyService = idempotencyService;
this.paymentService = paymentService;
}
@KafkaListener(topics = "payment.commands", groupId = "payment-processor")
public void onPaymentCommand(
@Payload PaymentCommand command,
@Header(KafkaHeaders.RECEIVED_TOPIC) String topic,
Acknowledgment ack) {
// Use the command's business ID as the idempotency key
// This key is stable across redeliveries of the same Kafka message
String idempotencyKey = "kafka:" + topic + ":" + command.getCommandId();
String requestHash = IdempotencyService.hashRequest(command.toString());
try {
Optional<IdempotencyRecord> existing = idempotencyService.findOrCreate(
SYSTEM_TENANT, idempotencyKey, requestHash
);
if (existing.isPresent()) {
log.info("Skipping duplicate Kafka message [commandId={}]",
command.getCommandId());
ack.acknowledge(); // Acknowledge to prevent redelivery loop
return;
}
// Process the payment command
paymentService.processCommand(command);
idempotencyService.markCompleted(SYSTEM_TENANT, idempotencyKey, 200, command);
ack.acknowledge();
} catch (Exception e) {
log.error("Failed to process payment command [commandId={}]",
command.getCommandId(), e);
// Do NOT acknowledge — allow Kafka to redeliver
// The PROCESSING record will block duplicate processing attempts
// Consider a dead-letter topic for repeated failures
}
}
}
The Transactional Outbox Pattern
One of the most important idempotency patterns in event-driven systems is the Transactional Outbox. It solves the dual-write problem: you cannot atomically write to a database and publish to a message broker in the same transaction.
// Without outbox: DANGEROUS — database write succeeds, broker publish may fail
@Transactional
public PaymentResult processPayment(PaymentCommand command) {
Payment payment = paymentRepository.save(buildPayment(command)); // ✓ persisted
kafkaTemplate.send("payment.events", buildEvent(payment)); // ✗ may fail silently
return PaymentResult.from(payment);
}
The outbox pattern writes the event to a database table within the same transaction, then a separate relay process publishes it to the broker:
// With outbox: SAFE
@Transactional
public PaymentResult processPayment(PaymentCommand command) {
Payment payment = paymentRepository.save(buildPayment(command));
// Written atomically with the payment record
OutboxEvent event = OutboxEvent.builder()
.aggregateType("Payment")
.aggregateId(payment.getId().toString())
.eventType("PaymentProcessed")
.payload(objectMapper.writeValueAsString(PaymentProcessedEvent.from(payment)))
.build();
outboxRepository.save(event);
return PaymentResult.from(payment);
}
// Separate relay: polls the outbox and publishes to Kafka
@Scheduled(fixedDelay = 500)
@Transactional
public void relayOutboxEvents() {
List<OutboxEvent> pending = outboxRepository.findByPublishedFalseOrderByCreatedAtAsc();
for (OutboxEvent event : pending) {
kafkaTemplate.send(resolveTopicFor(event.getAggregateType()), event.getPayload());
event.setPublished(true);
outboxRepository.save(event);
}
}
The Kafka message may be published more than once (if the relay crashes after publishing but before marking it published = true), but the consumer's idempotency check handles that.
Webhook Idempotency
When receiving webhooks from payment providers (Stripe, Adyen, PayPal), treat each webhook delivery as a potentially duplicate event:
@PostMapping("/webhooks/payment-provider")
public ResponseEntity<Void> receiveWebhook(
@RequestHeader("Webhook-Id") String webhookId,
@RequestHeader("Webhook-Signature") String signature,
@RequestBody String rawBody) {
// 1. Verify the webhook signature (ALWAYS do this first)
if (!webhookVerifier.verify(rawBody, signature)) {
return ResponseEntity.status(HttpStatus.UNAUTHORIZED).build();
}
// 2. Use the provider's webhook ID as the idempotency key
String requestHash = IdempotencyService.hashRequest(rawBody);
Optional<IdempotencyRecord> existing = idempotencyService.findOrCreate(
"webhook-consumer", webhookId, requestHash
);
if (existing.isPresent()) {
log.info("Ignoring duplicate webhook delivery [webhookId={}]", webhookId);
return ResponseEntity.ok().build(); // Return 200 to stop redelivery
}
// 3. Process the event
try {
WebhookEvent event = objectMapper.readValue(rawBody, WebhookEvent.class);
webhookProcessor.process(event);
idempotencyService.markCompleted("webhook-consumer", webhookId, 200, null);
return ResponseEntity.ok().build();
} catch (Exception e) {
log.error("Webhook processing failed [webhookId={}]", webhookId, e);
// Return 5xx to trigger redelivery from the provider
return ResponseEntity.status(HttpStatus.INTERNAL_SERVER_ERROR).build();
}
}
Edge Cases and Failure Modes
The Stuck PROCESSING Record
If the application crashes while processing a payment (after claiming the key but before finalizing), the record stays in PROCESSING state indefinitely. Subsequent retries from the client will receive a 409 Conflict.
Solution: Implement a timeout-based recovery job.
@Scheduled(fixedDelay = 60_000) // every minute
@Transactional
public void recoverStuckRecords() {
Instant staleThreshold = Instant.now().minus(Duration.ofMinutes(5));
List<IdempotencyRecord> stuck = repository.findByStatusAndUpdatedAtBefore(
IdempotencyRecord.Status.PROCESSING, staleThreshold
);
for (IdempotencyRecord record : stuck) {
// Check if the underlying payment was actually completed
Optional<Payment> payment = paymentRepository.findByIdempotencyKey(record.getIdempotencyKey());
if (payment.isPresent()) {
// Payment completed — update the record retroactively
idempotencyService.markCompleted(
record.getMerchantId(),
record.getIdempotencyKey(),
200, PaymentResponse.from(payment.get())
);
} else {
// Payment not found — mark as failed, allow client to retry with a new key
idempotencyService.markFailed(
record.getMerchantId(),
record.getIdempotencyKey(),
500, Map.of("error", "Processing timeout — please retry with a new idempotency key")
);
}
}
}
Concurrent Duplicate Requests
The UNIQUE constraint on (merchant_id, idempotency_key) handles races at the database level. However, ensure your service does not hold application-level locks that could cause deadlocks. The DataIntegrityViolationException catch in findOrCreate() is the correct fallback — catch it, query the existing record, and return it.
Different Request Bodies, Same Key
This is a client bug, not a server concern. The server should reject it with a clear error. Use the request_hash column to detect this:
HTTP/1.1 422 Unprocessable Entity
{
"error": "IDEMPOTENCY_KEY_CONFLICT",
"message": "The Idempotency-Key was previously used with a different request body. Generate a new key for this operation."
}
Caching Infrastructure Failures vs Business Failures
A critical distinction:
Business failures (payment declined, insufficient funds, invalid card) → cache the failure, return the same error on retry
Infrastructure failures (database timeout, downstream service unavailable, 5xx from processor) → do not cache, leave the record in PROCESSING or delete it, allow the client to retry
This distinction determines whether the idempotency record is finalized or left open for retry.
Observability and Testing
Metrics to Instrument
@Component
public class IdempotencyMetrics {
private final MeterRegistry meterRegistry;
public void recordNewRequest(String merchantId) {
Counter.builder("idempotency.requests")
.tag("type", "new")
.tag("merchant", merchantId)
.register(meterRegistry).increment();
}
public void recordDuplicate(String merchantId, String status) {
Counter.builder("idempotency.requests")
.tag("type", "duplicate")
.tag("cached_status", status)
.tag("merchant", merchantId)
.register(meterRegistry).increment();
}
public void recordConflict(String merchantId) {
Counter.builder("idempotency.conflicts")
.tag("merchant", merchantId)
.register(meterRegistry).increment();
}
}
Key metrics to track:
Metric
Alert Threshold
idempotency.requests{type=duplicate}
Spike > 5% of total may indicate client retry storms
idempotency.conflicts
Any nonzero value indicates client bugs
idempotency.records{status=PROCESSING} age > 5 min
Recovery job may be failing
DB table row count
Growth rate indicates cleanup job failures
Integration Test: Verifying Idempotent Behavior
@SpringBootTest
@AutoConfigureMockMvc
class PaymentIdempotencyTest {
@Autowired private MockMvc mvc;
@Autowired private IdempotencyRepository idempotencyRepository;
@Autowired private PaymentRepository paymentRepository;
@Test
void sameIdempotencyKey_shouldReturnSameResponseAndChargeOnce() throws Exception {
String idempotencyKey = UUID.randomUUID().toString();
String requestBody = """
{ "amount": 1000, "currency": "USD", "customerId": "cust_123" }
""";
// First request — should succeed and create a payment
MvcResult first = mvc.perform(post("/v1/payments")
.header("Idempotency-Key", idempotencyKey)
.header("Authorization", "Bearer test-merchant-token")
.contentType(MediaType.APPLICATION_JSON)
.content(requestBody))
.andExpect(status().isCreated())
.andExpect(header().string("Idempotency-Replayed", "false"))
.andReturn();
// Second request with same key — must not create a second payment
MvcResult second = mvc.perform(post("/v1/payments")
.header("Idempotency-Key", idempotencyKey)
.header("Authorization", "Bearer test-merchant-token")
.contentType(MediaType.APPLICATION_JSON)
.content(requestBody))
.andExpect(status().isCreated())
.andExpect(header().string("Idempotency-Replayed", "true"))
.andReturn();
// Response bodies must be identical
assertThat(first.getResponse().getContentAsString())
.isEqualTo(second.getResponse().getContentAsString());
// Exactly one payment must exist in the database
long paymentCount = paymentRepository.countByCustomerIdAndIdempotencyKey(
"cust_123", idempotencyKey
);
assertThat(paymentCount).isEqualTo(1);
}
@Test
void differentBodySameKey_shouldReturnConflictError() throws Exception {
String idempotencyKey = UUID.randomUUID().toString();
mvc.perform(post("/v1/payments")
.header("Idempotency-Key", idempotencyKey)
.contentType(MediaType.APPLICATION_JSON)
.content("""{ "amount": 1000, "currency": "USD" }"""))
.andExpect(status().isCreated());
mvc.perform(post("/v1/payments")
.header("Idempotency-Key", idempotencyKey)
.contentType(MediaType.APPLICATION_JSON)
.content("""{ "amount": 9999, "currency": "EUR" }""")) // Different!
.andExpect(status().isUnprocessableEntity())
.andExpect(jsonPath("$.error").value("IDEMPOTENCY_KEY_CONFLICT"));
}
}
Summary Checklist
Use this checklist when reviewing a payment API for idempotency correctness:
Idempotency Key Design
[ ] Keys are required for all mutating endpoints (POST, PATCH, DELETE with side effects)
[ ] Keys are scoped to a security boundary (merchant ID, tenant ID)
[ ] Key format is validated and length-capped server-side
[ ] Key generation strategy is documented for clients (UUID v4 or deterministic)
[ ] Key retention/expiry policy is defined and communicated in the API spec
Server-Side Deduplication
[ ] Idempotency records are stored in a durable store (not in-memory cache)
[ ] A UNIQUE constraint on (tenant, key) prevents race conditions at the DB level
[ ] Race conditions are handled by catching DataIntegrityViolationException and re-reading
[ ] Request body is hashed and compared to detect key misuse
[ ] Business failures are cached; infrastructure failures are not
Delivery Semantics
[ ] At-least-once delivery is assumed for all retry paths
[ ] Kafka consumers and webhook handlers implement idempotent processing
[ ] The Transactional Outbox pattern is used where dual-write is required
Operations
[ ] Stuck PROCESSING records have a timeout-based recovery job
[ ] Expired records are purged by a scheduled cleanup job
[ ] Duplicate rate, conflict rate, and stuck record count are monitored
[ ] Idempotency-Replayed: true header is present on cached responses for client debugging
Payment systems demand exceptional reliability. Idempotency is not a feature to add later — it is a fundamental correctness guarantee that must be designed in from the start. The patterns above, applied consistently, reduce duplicate payment incidents to near zero without sacrificing system throughput or developer ergonomics.
