Skip to content

The lab · how I think

If it ain't broke, break it.Then make it better.

This is a live model of a request-processing backend: a load balancer, three Go API nodes, Redis as a cache and Postgres as the source of truth. It fails the way real systems fail — and it's defended the way I'd defend one in production.

Pick an incident for a guided walkthrough, or inject faults yourself and switch defences off to see what each one is really doing. Nothing here is scripted: everything you see comes from a few simple rules, explained further down.

incident · The database gets slow

1/4Postgres slows down by 400ms

Only about 15% of requests miss the cache and reach Postgres — but each of those now holds an API worker for 400ms+ instead of ~10ms.

Timeouts cap every query at 250ms. Once more than half of queries are timing out, the circuit breaker trips open.

Watch: the breaker on the Postgres link

Each step changes the live system. Take your time — the simulation keeps running.

all systems operationalfaults: postgres slowt+00:00

system

click any component to inspect it
clients300 req/slb3 in poolapi-1queue 0api-2queue 0api-3queue 0redis85% hit ratepostgres—0% of 260 q/sbreaker closed
busy worker idle worker queuedashes = traffic · red = failing link

inject faults

api nodes
traffic
postgres
redis

defences

live metrics

last 18s
availability
100%good = succeeded in under 1s
p99 latency
—
good req/s
—
load amplification
—(arrivals + retries) ÷ arrivals

request outcomes

    latency distribution (this tick, 400 requests)

    <10ms25501002505001s2.5s5s5s+

    Waiting for traffic…

    event log

    1. Nothing has broken yet. Go on.

    How it works

    Four ideas behind every outage

    These four rules produce everything the lab does. The charts read from the same live simulation running above.

    01

    Little's law: slow is expensive

    capacity = workers ÷ time per request

    Each API node has 6 workers. A request holds one for its own 20ms of work plus every millisecond it waits on Redis or Postgres. That's why a slow dependency is so dangerous: it doesn't fail anything, it just quietly shrinks capacity.

    Flip it around and you get the number of requests in flight: arrival rate × time. Once that exceeds the worker count, the rest wait in a queue.

    100 req/s × 22ms = 2.2 in flight
    time per request now22ms
    capacity per node now268 req/s

    02

    Queueing: there is no “a bit slower”

    waiting ∝ ρ ÷ (1 − ρ)

    ρ is utilisation. At 50% busy, requests wait 1 unit; at 90%, 9; at 99%, 99. Latency stays flat for a long time, then hits a cliff. That's why you shed load at 90%, not 100%.

    shed at 90%now 38%0%50%100%

    03

    Circuit breakers: fail fast, recover safely

    closed → open → half-open → closed

    When more than 50% of Postgres calls fail, the breaker opens and stops calling it entirely — serving stale cached copies instead. After 4s it goes half-open and lets 10% of traffic probe. Healthy probes close it; failing ones re-open it.

    >50% failafter 4sprobes okclosedopenhalf-open

    04

    Retries: helpful, until they aren't

    load = arrivals × (1 + retry rate), up to 3×

    A retry turns a transient failure into a success — great when one node blips. But when failures come from overload, every retry is extra load on an already overloaded system. With 3 attempts, a struggling system can see 3× its normal traffic. Retry budgets, backoff with jitter, and load shedding stop that spiral.

    load amplification now1.00×
    1× arrivals3× max

    For real

    Where I've done this in production

    The lab is a toy. These weren't.

    Like how this thinks?

    See the work behind it, or get in touch.

    The Lab — how Richie thinks about reliability