Production-Grade Resilience for an API Migration Layer

Our client, a compliance technology provider serving regulated financial institutions, was moving customer teams from an older API onto a newer one. To make that move easy, they built a compatibility layer in between, so a team could migrate without rewriting its integrations against the new API. The layer was put together quickly, as a proof of concept, to show the migration path worked.

It worked, and real traffic started flowing through it. That’s when the usual thing happened with anything built fast to prove a point: it wasn’t tuned for the load it was now carrying. Response times under real traffic, and recovery when a downstream call failed, were rougher than a production path should be. The client asked us to take that layer the rest of the way, to production quality.

We measured first, then hardened the layer one change at a time:

• Under a stress test that injected failures into every downstream call, request success rose from 37% to 99.6%, proven by fault-injection testing on the client’s own staging.
• Interrupted requests now resume from the last successful step instead of starting over, with no duplicate work, moving the layer from POC quality to production quality.
• The client kept the load-and-failure test setup to re-run the whole analysis themselves as traffic grows, with no dependency on us to re-measure.

We brought a method with the hands: measure, change one thing, prove it, hand it back runnable.

Established a measured baseline before changing anything

• Ran repeatable load tests on the client’s own k6 and Grafana setup, so results were judged on their numbers, not ours.
• Found that scaling out wasn’t the fix: six pods performed about the same as one at low load, which pointed at the code, not the capacity.
• Set percentile targets (P50 through P99) across a range of throughput levels to measure every later change against.

Unified row-level security to stop repeated database checks

• Found the same permission checks running several times within a single request, each one hitting the database.
• Consolidated those checks into one place lower in the stack, so the check runs once per request.
• Kept the same access guarantees while removing the duplicate calls higher up.

Finished a parked optimization and tuned autoscaling

• Took an earlier orchestration-layer optimization that had been started and shelved, and brought it to production quality with proper test coverage.
• Tuned the database connections each pod holds, and the signals that trigger new pods, so capacity tracks real traffic instead of guessing at it.

Added two levels of retry so requests recover instead of restarting

• In-request retries the layer fires itself when a downstream call fails.
• Client-resend retries that resume from the last successful step, rather than re-running work that already went through.
• Recovery adds latency only when it actually fires. On a healthy request there is no added cost.

Made every step idempotent so retries stay safe

• A repeated request can’t create duplicate work.
• A retry with a changed payload is rejected rather than half-applied.
• Steps that already completed keep their original identifiers instead of running a second time.

Proved recovery on the client’s own environment

• Took down each downstream service in turn on live staging; the workflow failed cleanly, and a retry drove it to completion with no duplicates.
• Injected failures into every downstream call at a 25% rate, far past anything a healthy system would see, and measured completion: 37% without the retry work, 99.6% with it.
• Under that same stress test, p99 moved from 1.55s to 2.84s, the deliberate cost of recovering a request instead of losing it.

The layer now runs at production quality instead of POC quality, with recovery proven step by step on the client’s own environment. Because we left the test setup behind, the team can re-measure and re-tune on its own as the platform keeps changing.