Order Retry Logic That Prevents Duplicate Charges

Retry only transient failures: server errors (500, 502, 503, 504) and rate limits (429). Require an idempotency key on every state-changing request. Use exponential backoff with full jitter, cap total attempts, and escalate when retries exhaust. Replay the entire database transaction in a fresh transaction rather than retrying a single statement, and lean on upsert logic, advisory locks, or the outbox pattern to keep side effects from firing twice.

The business risk behind order retry logic is simple: a customer gets charged twice, or an order silently vanishes. Both outcomes trace back to the same technical gap, retrying a mutation without a durable way to recognize "I already did this." Idempotency keys close that gap. Transaction-scoped retries close the other half, ensuring a retried operation either commits cleanly or leaves no partial trace. The Azure Retry pattern and Polly in .NET both codify this thinking, and platforms like SafeFly apply the same logic when mirroring trades across accounts, where a duplicated order is not a minor bug but a real financial exposure.

Retry: 500, 502, 503, 504, 429
Never retry: 400, 404, 422 (fix the request, don't repeat it)
Always attach: an idempotency key on any request that changes state

Quick reference: Klarna's escalation policy recommends escalating immediately when a request arrives without an idempotency key, rather than retrying blind.

Key Takeaways

Order retry logic works reliably only when idempotency keys, transaction-scoped replay, and capped exponential backoff with jitter are implemented together, not separately.

Point	Details
Classify errors first	Retry 5xx and 429 only; treat 400, 404, and 422 as fixes the caller must make.
Require idempotency keys	Cache the response with Redis, back it with a database upsert, and reject requests missing the key.
Use full jitter backoff	Randomize each wait between zero and the exponential delay to avoid synchronized retry storms.
Replay whole transactions	Never retry a single statement; roll back and reissue the entire transaction on SQLSTATE 40001.
Cap attempts and escalate	Set a hard retry limit, route exhausted attempts to a DLQ, and escalate immediately when idempotency is missing.

Which Errors Are Safe to Retry in Order Processing Retries?
How Should You Structure Backoff and Retry Pacing?
What Role Do Idempotency Keys Play in Failed Order Recovery?
How Do Transactions Behave Correctly Under Retry?
What Database Patterns Prevent Duplicate Orders?
How Do You Know Your Retry Policy Actually Works?
When Should Failed Orders Escalate Instead of Retry?
How Does SafeFly Apply Order Retry Logic in Production?
What Does the Evidence Actually Say About Retry Logic?
Sources

Which Errors Are Safe to Retry in Order Processing Retries?

Retry decisions come down to one question: will trying again change the outcome? Server-side failures answer yes. Client-side failures almost always answer no.

Retryable HTTP statuses include 500 (internal server error), 502 (bad gateway), 503 (service unavailable), 504 (gateway timeout), and 429 (rate limited). These signal a temporary condition on the server or network, not a problem with the request itself. Industry guidance on resilient checkout flows draws this line clearly: retries belong on transient 5xx errors and rate limits, never on business-logic rejections.

Non-retryable statuses are 400 (bad request), 404 (not found), and 422 (unprocessable entity). Retrying these wastes attempts and delays the real fix, surfacing the validation error, correcting a malformed payload, or pointing the caller at a missing resource.

Database errors need their own classification for order transaction failures:

Retryable: SQLSTATE 40001 (serialization failure) and deadlock conditions, both symptoms of contention that usually clears on a fresh attempt.
Not retryable: SQLSTATE 23505 (unique constraint violation). This code often means your own retry already succeeded once, so treat it as a signal to check for an existing record, not a reason to try again.

The hardest case is the "unknown outcome," where a request times out and you have no idea whether the order was created. Without an idempotency key, you cannot safely retry an unknown outcome. With one, the retry becomes a lookup instead of a gamble.

How Should You Structure Backoff and Retry Pacing?

Fixed-interval retries hammer a struggling dependency at a predictable rhythm, which is exactly the wrong move when that dependency is already overloaded. Exponential backoff spreads attempts out, doubling the wait after each failure so a system under load gets breathing room instead of a steady drumbeat of repeat requests.

Jitter fixes the next problem: if a thousand clients all back off on the same schedule, they retry in sync and create a new spike. Full jitter solves this by randomizing the wait between zero and the calculated exponential delay, random(0, exponentialDelay), so retries fan out instead of clustering.

Attempt 1 fails, wait roughly 0 to 1 second.
Attempt 2 fails, wait roughly 0 to 2 seconds.
Attempt 3 fails, wait roughly 0 to 4 seconds.
Attempt 4 fails, wait roughly 0 to 8 seconds, then stop and escalate.

That four-attempt schedule works for most order APIs, but the right ceiling depends on how critical the dependency is. A payment gateway call inside a checkout flow should stay aggressive but short, capping the total retry window under 30 seconds so the customer isn't staring at a spinner. A background reconciliation job talking to a less critical service can afford a longer window measured in minutes. Polly's retry strategy documentation offers configurable delay generators that implement this exact pattern in .NET without hand-rolled math.

Pro Tip: Set your total retry window shorter than the caller's own timeout. A retry loop that outlives the client's patience just produces a duplicate request from a caller who already gave up and tried again.

How Should You Structure Backoff and Retry Pacing? — overview diagram

What Role Do Idempotency Keys Play in Failed Order Recovery?

An idempotency key turns a risky retry into a safe one by guaranteeing that the same key always returns the same durable response, no matter how many times the request repeats. The server must cache or persist that response, not just detect the duplicate, because a fresh attempt has to get back the original order confirmation, not a new one.

The reliable implementation runs on two layers. A fast-path cache, commonly Redis with a SET ... NX command, catches most duplicate requests within milliseconds and returns the stored result immediately. Underneath it sits an authoritative database check, an upsert against a durable idempotency table, that catches anything the cache missed because of eviction, a restart, or a race condition. Never let the cache alone be the gatekeeper for a purchase retry mechanism; a cache miss should fall through to the database, not to a fresh charge.

Key lifetime matters more than most teams assume. PostGrid's API documentation specifies a 24-hour idempotency key validity window, and production systems commonly extend that to 24 to 48 hours to cover slow retries and delayed webhooks. Validation should enforce a consistent key format, require the header on every state-changing request, and reject malformed keys outright rather than silently generating one.

Three failure modes deserve explicit handling:

Missing key on a request that requires one: reject with a 422, don't guess.
Invalid key format: reject immediately, before touching the database.
Persistence failure (the key can't be written): escalate to a human process instead of continuing to retry blind.

How Do Transactions Behave Correctly Under Retry?

A retry that touches only part of a failed transaction is more dangerous than no retry at all. The rule is absolute: always replay the entire transaction in a fresh transaction, never retry a single statement inside one that already aborted.

Roll back completely on a serialization failure or deadlock, then start over from the first statement.
Treat the retried transaction as a pure function with no side effects outside its own boundary; anything that reaches an external system (an email, a webhook, a payment call) belongs outside the transaction or routed through an outbox.
Use CockroachDB's retry savepoints, or an explicit client-side loop that catches SQLSTATE 40001 and reissues the whole transaction, as YugabyteDB's documentation on transaction retries recommends.
Match isolation level to risk: SERIALIZABLE for financial ledger writes where correctness overrides throughput, READ COMMITTED with SELECT FOR UPDATE for lower-risk flows where a targeted lock is enough.

Framework magic makes this harder than it looks. Combining @Retryable and @Transactional in Spring runs into ordering problems in the AOP proxy chain, where the retry can wrap the wrong transaction boundary and quietly retry inside an already-committed or already-aborted context, a pitfall documented in Baeldung's guide to Spring retry and transactions. A programmatic TransactionTemplate paired with a RetryTemplate sidesteps the ambiguity by guaranteeing a genuinely new transaction on every attempt.

Pro Tip: If your ORM logs a successful commit after a "failed" transaction retried, you have an isolation bug, not a network bug. Check your isolation level before you touch the retry code.

What Database Patterns Prevent Duplicate Orders?

Three patterns, used together, cover almost every duplicate-order scenario that retries create.

Advisory locks. pg_try_advisory_xact_lock serializes attempts tied to the same idempotency key without taking a table-level lock, so two retries racing on the same key can't both proceed at once. One wins the lock and processes; the other waits or backs off.
Upsert idempotency table. A dedicated table tracks each key with a status column (PENDING, COMPLETED) and a response_payload column. An INSERT ... ON CONFLICT statement detects a duplicate key at the database level, the same layer DistributedRequest's guide to wrapping database transactions for safe retries recommends for eliminating race windows between the key check and the mutation.
Outbox pattern. The event row that triggers a downstream side effect, a notification, a webhook, a ledger update, gets written inside the same transaction as the order mutation. A separate relay process reads the outbox and delivers the event, so a crash between committing the order and sending the notification can't produce a lost or duplicated message.

The cleanest implementation wraps all three in one transaction: check or insert the idempotency key, take the advisory lock, apply the business mutation, write the outbox row, commit. Nothing outside that boundary needs to be retried.

Redis cache misses still happen, and the fallback matters as much as the primary path. When the fast-path cache doesn't have the key, fall through to the authoritative database check rather than treating the miss as a green light for a fresh order. Track a cache_miss_rate counter so a spike in misses (from a Redis restart or an eviction storm) shows up in observability before it shows up as duplicate orders in support tickets.

Pro Tip: Test your ON CONFLICT clause under real concurrency, not just in a single-threaded unit test. Race conditions in idempotency logic almost never show up until two requests actually arrive within milliseconds of each other.

How Do You Know Your Retry Policy Actually Works?

A retry policy you haven't stress-tested is a guess wearing a config file. Measure it with a specific set of metrics: retry_count per request, deduplication_hit_rate (how often the idempotency layer catches a repeat), cache_miss_rate, deadlock_count, outbox_delivery_rate, and p95 latency measured both with and without retries active, since a retry policy that fixes reliability but triples tail latency has just traded one problem for another.

Simulate a gateway failure rate in the 50 to 60% range and confirm the system recovers the transaction without producing a duplicate charge.
Load-test the idempotency cache under concurrent duplicate requests, not just sequential ones.
Watch for a thundering herd signature, a synchronized spike in retry traffic right after a dependency recovers, which usually means jitter isn't wide enough.

A NestJS resilient checkout implementation documented improved recovery in degraded scenarios once auto-tuned timeouts and retry counts replaced static values, showing that retry logic tuned against real telemetry outperforms a fixed schedule copied from a blog post.

Set alert thresholds on the metrics above rather than on raw error rates alone, and write a short runbook entry for each one: what the spike means, what to check first, and when to flip a circuit breaker instead of continuing to retry.

When Should Failed Orders Escalate Instead of Retry?

Retrying forever is not a policy, it's a way to hide a problem until it's bigger. A workable operational policy sets hard limits and a clear next step when those limits are hit.

Retry cap: four to six attempts for latency-sensitive checkout flows, capped under 30 seconds total; background reconciliation jobs can extend to a longer window measured in minutes.
Dead-letter queue (DLQ): route exhausted retries here for asynchronous processing, not for requests where the customer is waiting live.
Manual retry endpoint: give operators an admin path to reprocess a DLQ entry once the underlying issue (a gateway outage, a bad deploy) is confirmed resolved.
Immediate escalation, no retry loop: when a request arrived without an idempotency key, or when the outcome is genuinely unknown and unsafe to guess at, Klarna's escalation and retry policy recommends skipping automated retries entirely and pulling in a human, with an example schedule of waiting 5 seconds, then 5 minutes, then 5 hours before escalation.

The decision tree is short: retryable error and valid idempotency key, retry with backoff. Retries exhausted, send to DLQ. No idempotency key or unresolvable ambiguity, escalate now.

How Does SafeFly Apply Order Retry Logic in Production?

Mirroring a trade from a lead account to several follower accounts is, structurally, the same problem as retrying an order: one action has to land exactly once per account, even when a connection drops mid-execution. SafeFly applies idempotency keys per mirrored trade instance, so a retried mirror attempt returns the original execution result instead of placing a second position.

Broker-side protective stops mean a failed retry never leaves a position naked; the stop lives on the broker, not the client connection, so a disconnection during a retry doesn't remove downside protection.
Outbox-style event persistence fits naturally into the mirroring pipeline, recording each mirrored trade as a durable event before confirming it downstream.
Daily profit and loss lockouts feed the escalation logic, tightening retry tolerance automatically once a loss threshold is near.

Pro Tip: If you're managing multiple Tradovate accounts by hand, ask whether your current process could survive a dropped connection mid-trade without duplicating or missing an order. SafeFly's how it works page walks through the mirroring and stop-placement mechanics in detail, and the pricing page covers plan tiers for traders running more than one account.

What Does the Evidence Actually Say About Retry Logic?

Most retry advice online treats backoff and jitter as the hard part. They're not. The math is settled and every serious library gets it right. The part teams actually get wrong is idempotency, specifically, assuming a cache check is enough and skipping the authoritative database upsert underneath it. That shortcut works fine in testing and fails exactly when it matters, during a Redis restart under load.

The transaction-scoping question gets underrated too. Retrying a statement inside an already-aborted transaction is a common bug, and framework-level @Transactional and @Retryable annotations make it easy to introduce without noticing, because the ordering failure doesn't throw an obvious error.

If you're building this from scratch, sequence the work: idempotency contract first, transaction replay second, backoff tuning last. Backoff tuning is the part you can safely improve after launch. Missing idempotency is the part that generates a support ticket about a duplicate charge.

Sources

Start with the Azure Retry pattern for the architectural overview, then Polly's retry strategy docs for a concrete .NET implementation. For database-level retry semantics, YugabyteDB's transaction retry guide and DistributedRequest's transaction-wrapping walkthrough cover the code-level detail this article summarizes. PostGrid's API documentation and Klarna's escalation policy both document production idempotency and escalation rules from payment-adjacent platforms. For a full working example, the NestJS resilient checkout write-up is worth reading end to end, as is CryptoPayr's piece on maintaining checkout throughput when card payments fail.