The moment a connection drops, pause every automated order and mirroring action across all accounts. Pull a fresh REST snapshot of positions, open orders, and fills for each Tradovate account before trusting any streamed data. If a venue shows a sequence-ID gap outside its normal backlog window, discard the local state for that venue and force a full resnapshot. Otherwise, apply the missed deltas and resume.
TL;DR:
- Pull a full REST snapshot of positions, open orders, and fills immediately after any connection drop, before trusting streamed data.
- Verify sequence IDs during reconnection; discard local state and fetch a new snapshot if there is a sequence gap or offset no longer exists in the provider's backlog.
- Conduct position comparisons with both absolute and relative thresholds, requiring multiple failed checks before forcing a resnapshot to prevent unnecessary resets.
- Large or persistent sequence gaps, missing history, or outrunning the backlog window require a complete rebuild rather than a partial resync to ensure data integrity.
- Test resync procedures regularly with simulated outages, schedule rebuilds during low-impact times, and document root causes to reduce the risk of critical errors during real incidents.
Table of Contents
- What Is the Correct Order for Position Resync After Outage?
- How Do You Safely Resync Positions Step by Step?
- How Do You Verify and Reconcile Position Differences?
- When Do You Need a Full Rebuild Instead of a Resync?
- Which Safeguards Prevent the Next Resync Error?
- How Should You Test Your Resync Process Before You Need It?
- Why Conservative Resync Rules Matter for Multi-Account Traders
- How SafeFly Reduces Resync Risk Across Your Accounts
- Sources
- FAQ
What Is the Correct Order for Position Resync After Outage?
Getting the sequence wrong is how a five-minute outage turns into a five-figure mistake. The correct order for resyncing positions after outage protects you from acting on stale or incomplete data during the most dangerous ten to thirty minutes of any incident.
- Freeze automation. Pause mirroring and any bot that sends outbound orders across all connected Tradovate accounts. No new positions, no adjustments, until state is verified.
- Check the source, not your terminal. Confirm exchange and broker operational status pages and REST health endpoints before assuming the problem is local.
- Pull authoritative snapshots. Fetch REST-based positions, open orders, balances, and recent fills for every account. This is your source of truth, not the last cached values your terminal held before the drop.
- Compare net exposure per instrument. Line up what each account should be holding against what REST reports. Log every mismatch, no matter how small.
- Hold the gate. Keep automated execution blocked until reconciliation clears or a human signs off manually.
Skipping step three is the most common failure. Traders reconnect, see a familiar-looking chart, and assume the position ladder is accurate. It often isn't.
How Do You Safely Resync Positions Step by Step?
Resyncing positions after failure is not a single action. It's a sequence with checkpoints, and skipping a checkpoint is how duplicate orders and phantom fills get created.
- Reconnect with backoff. Don't hammer the API immediately after a drop. Use exponential backoff, then resubscribe to your data streams once the connection stabilizes. Do not resume stateful processing yet.
- Check the sequence ID on the first incoming delta. When the first update arrives, verify that its
prevSeqor sequence identifier is contiguous with what your stored snapshot expects. The correct recovery sequence after a WebSocket drop is to detect the disconnect, reconnect with backoff, and fetch a full REST snapshot before ever trusting the stream again. - Discard and refetch if the sequence breaks. If the ID isn't contiguous, the local state for that venue is unreliable. Clear it and pull a fresh REST snapshot rather than trying to patch around the gap.
- Use partial resync only when the numbers line up. Some providers support partial resynchronization through replication IDs or backlog windows. This approach works only when the offset your client needs still falls inside the provider's backlog. Redis handles this by replaying missed commands only when the replica's ID and offset match the primary's backlog; if the gap is too large, a full resync is the only safe option.
- Run reconciliation as a read-only pass first. Compare local state against the fresh snapshot without writing anything. Only escalate to corrective action when the deltas exceed your defined thresholds, and log every write with an operator identifier and timestamp for audit purposes.
Pro Tip: Never let your reconciliation logic double as your execution logic. Keep the read path and the write path in separate functions. A syncer that only reads is safe to run every few seconds; one that reads and writes indiscriminately can create the exact duplicate-order problem you're trying to prevent.
This sequence matters more with multiple accounts running in parallel, because a sequence break on one venue link doesn't necessarily mean the others broke too. Treat each account's stream independently, and never assume a clean reconnect on Account A means Account B is fine.
How Do You Verify and Reconcile Position Differences?
Verification is where most of the actual risk-reduction work happens, and it's the part traders rush past because it feels slower than just "getting back to trading."
- Compare your local cache's reported position against the REST-confirmed position for every instrument, on every account, individually. Aggregated totals hide single-account drift.
- Set both an absolute threshold (say, a fixed number of contracts) and a relative threshold (a percentage of expected size) before triggering an automatic resnapshot.
- Require multiple consecutive failed checks before forcing a resync. A single mismatched read can be a timing artifact rather than real drift. Case work on multi-venue state desync shows that reconciliation loops with consecutive-check gating prevent unnecessary resnapshot thrashing while still catching genuine divergence.
- When fills conflict, treat REST-confirmed fills as the deciding record. Flag anything your local log shows that REST doesn't, and vice versa.
- Keep a full audit trail: what diverged, by how much, on which account, and what correction was applied.
One detail traders manning several accounts consistently underestimate: drift on a single instrument across two accounts running the same strategy is often not random. It usually points to a mirroring delay or a stale order acknowledgment, not two unrelated glitches. Treat correlated drift as a single root cause investigation, not two separate tickets.
When Do You Need a Full Rebuild Instead of a Resync?
Not every outage calls for a full rebuild, but pretending a partial catch-up will work when it can't just delays the inevitable full reset, usually after more damage is done.
- Impossible offsets. If the sequence or offset your client needs no longer exists in the provider's backlog, a partial catch-up is off the table.
- Purged history errors. Errors resembling replication systems' "required data has been purged" condition mean the missing window is gone, not just delayed.
- Backlog too small for the outage duration. A short backlog window combined with a long outage almost guarantees you've outrun what partial resync can recover. MongoDB's own guidance for a replica set member that has fallen too far behind is to either wipe local data and let it re-sync from scratch, or copy a data directory from a known-good member.
- Safe rebuild steps. Empty the unreliable local store and bootstrap fresh from a REST snapshot, or, where a known-good replica exists, copy its recent state instead of rebuilding from zero.
- Timing and communication. Schedule the rebuild for a maintenance window when possible, take a backup before any destructive step, and tell your desk and risk manager the expected downtime before you start, not after.
Which Safeguards Prevent the Next Resync Error?
The best resync procedure is the one you rarely need. Building the right safeguards into your daily operation cuts both the frequency and the severity of the next incident.
- Run a reconciliation loop that operates independently of your streaming handler, comparing REST-confirmed state on a schedule rather than only during incidents. Independent reconciliation loops with venue-specific gating are what caught silent drift before it became large, unnoticed exposure in the multi-venue case work above.
- Keep protective stops enforced at the broker level, not just inside your trading software, so they stay active even if your connection drops. A disconnection during multi-account futures trading is exactly when software-side stops disappear along with the connection.
- Cap daily risk with automated profit and loss lockouts that trigger regardless of whether your monitoring tools are online.
- Connect through secure OAuth integrations rather than shared credentials, reducing the operational surface that can fail during a reconnect.
- Put these controls into your written incident runbook, and test them on a schedule instead of discovering gaps during a live event.
Pro Tip: If your protective stops live only inside your own trading application, they die with your connection. Broker-side stops are the difference between "I lost my internet for four minutes" and "I lost my internet for four minutes and my position was still protected."
How Should You Test Your Resync Process Before You Need It?
A resync procedure you've never rehearsed is a guess, not a plan. Testing turns your runbook into something you can trust under pressure.
- Simulate simultaneous multi-venue disconnects. Confirm your sequence checks actually force a resnapshot on each affected account rather than silently resuming from stale data.
- Automate the snapshot-and-reconcile cycle as read-only. Run it constantly in testing, and only allow corrective writes once you've confirmed the thresholds are tuned correctly.
- Time your rebuild scenarios. Know roughly how long a full rebuild takes for your account count so you can schedule disruptive tests, and real rebuilds, during low-impact windows.
- Run an after-action review every time. Document the timeline, the root cause, and whether your K-consecutive-check thresholds need adjusting based on what actually happened.
Why Conservative Resync Rules Matter for Multi-Account Traders
The temptation after any outage is to get back to trading as fast as possible. That instinct is exactly backward for anyone running mirrored positions across several Tradovate accounts. Silent drift on even one account can compound into exposure nobody notices until the damage is done. A few extra seconds spent verifying a REST snapshot costs you far less than an unverified resume that opens or closes the wrong position on the wrong account. The trade-off is never speed versus safety. It's brief, deliberate latency versus a mistake that outlives the outage that caused it.
— Arturo
How SafeFly Reduces Resync Risk Across Your Accounts
SafeFly is built around the same principle this guide argues for: correctness before speed when a connection breaks. Instead of manually rebuilding positions across accounts after a disconnect, SafeFly's automated mirroring keeps replicated accounts aligned to a lead account continuously, and its protective stops stay in force even when your connection drops, because they live with the broker, not inside a terminal that just went dark.
Daily profit and loss lockouts cap risk automatically during exactly the reconnect windows where drift is most likely to go unnoticed, and connections can run through secure OAuth integration rather than shared credentials sitting in a script. Trade analytics and AI coaching give you visibility into how accounts actually behaved during and after an incident, not just a gut sense that things "look fine now." See the full mechanics on the SafeFly how it works page, or start a trial and let broker-side stops handle the outage risk this guide just walked you through.
Sources
For deeper technical detail on the mechanics behind this guide, see the WebSocket reconnection sequence for trading bots, Redis partial resynchronization behavior, and exchange-side recovery and resync flows. For a broader view on resilience in trading infrastructure, institutional-grade tooling for execution and resilience covers similar ground from an infrastructure angle.
- Redis: How partial resynchronization works in Redis
- Crypto market making position drift: How multi-venue state desync defeats pre-trade-risk
FAQ
What Is the First Thing I Should Do After an Outage?
Pause all automated mirroring and outbound orders immediately, then pull a REST snapshot of positions and fills before trusting any streamed data.
How Do I Know If I Need a Partial or Full Resync?
Check whether the sequence ID or offset your account needs still falls inside the provider's backlog window; if it doesn't, or if you get an error resembling purged history, you need a full rebuild.
How Long Does a Full Position Rebuild Take?
It depends on account count and data volume, so schedule rebuilds during maintenance windows and communicate expected downtime to your desk in advance rather than assuming a fixed duration.
Can My Protective Stops Fail During a Disconnect?
Stops managed only inside a trading terminal can disappear when the connection drops; broker-side stops, like those SafeFly enforces, stay active independent of your connection status.
How Often Should I Test My Resync Procedure?
Test it on a regular schedule using simulated multi-venue disconnects, not just after a real incident, so your sequence checks and thresholds are proven before you actually need them.