Designing consistent sign-in rate limits across replicas
Three backend replicas split a five-attempt counter. I replaced per-instance state with shared, concurrency-safe counters without adding a new service.
Disclosure: Client and internal project details have been anonymized.
Overview
An e-commerce storefront needed to limit repeated sign-in attempts from the same source IP. After five invalid-credential failures, the system had to block further attempts for five minutes, return HTTP 429, and explain the delay in the sign-in interface.
The flow worked in development but behaved inconsistently in staging. I used deployment logs to show that the counter lived inside each backend replica—one of several running copies of the service—so five attempts could be split across independent maps.
Rather than introduce a new datastore, I redesigned the limiter around a shared record store already available to the backend. Versioned updates kept concurrent replicas consistent, while an explicit fail-open policy protected sign-in availability. The delivered behavior gave customers the intended five-attempt boundary and a clear explanation instead of letting replica routing decide whether the limit appeared.
The challenge
The limiter had to count only invalid credentials from the same source IP, lock after five failures, return HTTP 429 during the cool-down, and leave the existing authentication states unchanged. Locked users also needed a clear explanation in the sign-in interface.
The first version enforced the limit in the backend and showed locked users a dedicated banner. It checked the limiter before reCAPTCHA, avoiding an unnecessary upstream request for an already-locked IP.
Why the initial approach was reasonable
The first limiter used a module-level Map. This kept the algorithm synchronous and added no infrastructure dependencies.
It was explicitly a best-effort design because scaled or cold-started runtimes could split or reset counters.
For the initial scope, it enforced the limit without adding a shared datastore. In staging, the limitation appeared often enough that the design could not remain.
Finding the distributed-state failure
The expected banner did not appear in staging. Tracing the response from the backend into the sign-in interface showed that the wiring matched development.
Application logs provided the next clue. At least three backend replicas served the login action behind a Kubernetes service: three startup events for the current bundle appeared within roughly 250 milliseconds. The replicas also recycled periodically.
Before · process-local state
After · shared state
Each replica had its own map. Five consecutive attempts from one IP could be divided so that no replica reached five. Recycling a replica could also erase its partial count.
Choosing shared state without adding a new service
Moving the limiter state into commercetools Custom Objects gave every replica access to the same counters through a record store the backend already used. Redis was a possible future option, but the project had no Redis dependency or infrastructure. Of the evaluated options, Custom Objects were the only shared store that added no new service to operate.
Each source IP maps to a dedicated Custom Object containing the failed-attempt count, the start of the current window, and the lock expiration time when applicable.
The rate-limit rules remained separate from storage: pure functions calculate the next entry and decide whether it is locked, while a small asynchronous store contract handles shared persistence. This kept the decision logic independently testable.
Making shared updates safe under concurrency
Shared correctness required handling one additional case: two replicas could read the same counter version and update it at the same time.
Updates use commercetools' version-based optimistic concurrency. Each update includes the version returned by the previous read. If another replica has already changed it, the limiter reads the latest value, recalculates, and retries within a fixed bound.
Keeping rate limiting out of the critical failure path
Preserving authentication semantics
Only invalid credentials increment the counter; unrelated authentication failures do not.
Locked requests are rejected before reCAPTCHA and the downstream authentication call. A successful sign-in deletes the source IP's stored failure record.
SHA-256 converts the source IP into a fixed-length Custom Object key that satisfies the store's character rules. This choice solves storage compatibility, not anonymization. A keyed HMAC would serve a different privacy goal; for this design, the derived key remained sensitive data rather than being presented as anonymous.
Verification
The storage-independent tests exercised threshold and window behavior plus version conflicts in the read-modify-write loop, including bounded retry exhaustion. API tests covered the Custom Object adapter and its fail-open paths. This verified concurrency deterministically at the shared write-conflict boundary.
The deployed user-path check submitted five incorrect passwords and observed the expected rate-limit message.
Read or write failures that weaken protection are logged at error level; a failed cleanup is logged at warning level because it leaves a stale tally rather than disabling the limiter. Those severity levels make loss of protection distinguishable from cleanup failures in operational logs.
Accepted operating policy
One explicit operational tradeoff avoided introducing a new service: Custom Objects do not expire automatically, and the backend had no scheduled cleanup. Records not removed by a later successful sign-in can remain indefinitely, so the container has no hard size limit.
The choice traded automatic retention cleanup for consistent enforcement using infrastructure the team already operated. It does not weaken the shared five-attempt decision; it defines how limiter records are retained.
Outcome
Every backend instance now reads the same counter, so replica routing and pod recycling no longer divide the five-attempt decision. The deployed check confirmed the intended lockout message after five incorrect passwords, while successful sign-in and unrelated authentication errors retained their previous behavior.
The change restored a consistent customer-facing safeguard without adding a service for the team to deploy or operate. Shared, versioned counters enforce one decision across replicas, and storage failures remain visible without turning into a login outage.