Stop the operator panel freezing when you click through tabs

If you clicked through the operator panel quickly, the tab you asked for sometimes never arrived. The page sat there, the old content stayed on screen, and nothing said why. That was our bug, it was a silly one, and 2.2.0 fixes it. Here is what was happening and what changed underneath.

The freeze, and what caused it

The panel is a Next.js app, and every page in it is a dynamic server render that calls the API several times. One page view costs about a dozen API calls on its own. On top of that, Next prefetches links: every link in the viewport, again on hover, and again whenever a cache entry goes stale or a refresh empties the cache. A prefetch whose target sits behind a loading.tsx boundary renders every layout down to that boundary, API calls included.

Measured on a production build against a counting mock of the API, the first load of an end-user page fired 21 prefetch requests before the operator clicked anything. On /applications with three applications, five of the prefetches turned into a GET /applications/:id call, and that first load cost 13 API calls in all.

Those requests all counted against one bucket. The API rate limit was keyed on the client address, and in Docker the address every operator request arrives from is the panel container. So the whole team shared 100 requests a minute. One person working through a support ticket collected 429s within a few pages, and the panel of the day rendered a failed navigation as nothing at all. In a headless run with prefetching on, 5 of 27 tab clicks never committed.

Three changes fix it, and they are independent:

• Links do not prefetch. src/components/Link.tsx is now the panel's only link component and it defaults prefetch to false. A test fails the build if any other file reaches next/link, with one exception: the useLinkStatus hook, which the component that dims a clicked label imports. A click now fetches exactly one render.
• Every switch shows a loading state. The end-user tabs, the email sub-tabs and the account pages have loading.tsx boundaries, and the navigation dims the label you clicked from the moment you click it.
• The limiter counts per caller, which is the next section.

After the change, a second headless run committed all 36 of its tab clicks. It is a separate run, not the same clicks replayed. A busy API is also shown as busy now: a 429 or 503 renders a notice with a countdown and a bounded retry that honours Retry-After, rather than an empty list that reads like real data.

Per-caller rate limits

The global limiter now keeps one bucket per caller identity, picking the most specific identity the request has proved. A signed-in operator gets their own budget instead of sharing one with everyone behind the same address.

The window is RATE_LIMIT_WINDOW_MS, 60 seconds. The two authenticated budgets are optional settings: unset, each defaults to the larger of its own default and RATE_LIMIT_MAX, so a deployment that had raised that one knob does not lose headroom on upgrade. Two ceilings sit on top: all operators and end users seen from one address share 3000 per window (RATE_LIMIT_AUTHENTICATED_IP_MAX, API-key traffic exempt), and rejected credentials from one address are capped at RATE_LIMIT_AUTH_FAILURE_MAX, which defaults to RATE_LIMIT_MAX.

600 per operator is sized off the panel itself. One page view is about 12 API calls, plus up to about 22 route prefetches in a production build if you turn prefetching back on, so roughly 34. That is about 17 page views a minute, one every 3.5 seconds, for a whole minute, from one person. Auth endpoints keep their own tighter per-route caps either way.

Fewer database queries

Every signed-in operator request used to read the operator row, the session head and the membership before the handler ran. An application route added the application row, and a MEMBER two more for grants. The panel's application overview issues seven such requests per page view, so about half of its queries were re-reading the same rows.

Those reads are now cached inside each API process, and a warm request makes none of them. Measured against the same data:

Two more things stopped being expensive. Per-application stats are one query now, cached for 60 seconds in Redis under rk:stats:app:<id>, so the overview tile's counts can lag by up to a minute. Turning billing on or off drops the key and shows at once. Super-admin lists no longer query per row: the application list made 3 queries per listed application and the tenant list 5 per tenant. A sort on a computed value scans up to 500 rows, so that was up to 1,500 queries at once on the application list and up to 2,500 on the tenant list. Each is a fixed handful now whatever the page size, and the super-admin overview is one statement, cached for 60 seconds under rk:admin:overview.

Revocation is still immediate

Caching authentication state is the part worth reading twice, so here is exactly what it does. Only a successful read is cached. A missing operator, an ended session, a session stamp newer than the token, or a missing membership is never stored, so a refusal is always re-checked against the database.

Sign-out, sign out everywhere, a revoked session, a password change or reset, and any role, scope, grant or membership change drop the cached entry at once in the process that made the change, and publish the same invalidation over Redis so every other API replica drops it too. A read that raced an invalidation is discarded rather than stored. The session and grant caches serve nothing at all until the Redis subscription is confirmed, at boot and after every reconnect. One map is exempt: it holds only which workspace an application belongs to, and nothing ever changes that, so a lost message cannot make it wrong.

One window is left: a publish that fails while the publishing process still believes it is connected. That is what the TTL is for.

# How long another API replica can keep admitting a revoked
# session if the invalidation message was lost. Default 5000 ms.
OPERATOR_AUTH_CACHE_TTL_MS=5000

# Set 0 to turn the cache off and read every request from the
# database, as 2.1.x did.
OPERATOR_AUTH_CACHE_TTL_MS=0

On a single-process deployment there is nothing to lag: the process that revokes is the process that serves. The window applies to replicas, and only when the message did not arrive.

Redis is now Valkey 8.1

The redis service in every compose file, and in CI, now runs valkey/valkey:8.1-alpine instead of redis:7-alpine. The reason is licensing, not performance. The floating redis:7-alpine tag resolves to Redis 7.4, which is RSALv2/SSPLv1: source-available, not open source. Valkey is the Linux Foundation's BSD-3-Clause fork of Redis 7.2. Rekey is software you run yourself, so the default stack should be something you can actually run without reading a licence first. Valkey is protocol-compatible with the ioredis and BullMQ usage in this codebase, and no application code changed.

The service name, REDIS_URL and REDIS_PASSWORD are unchanged, and the URL still uses the redis:// scheme. The volume is new: rekey_valkey in docker-compose.yml and docker-compose.prod.yml, valkey-data in docker-compose.api.yml, replacing rekey_redis and redis-data. Valkey 8.1 loads a Redis 7.2 file but refuses a Redis 7.4 one, and most existing self-hosted volumes are already in the format it refuses, so an in-place swap would have been a boot-time outage on upgrade for most installs.

So the store starts empty. That is deliberate. This datastore is a cache, a queue and a lockout store, not a system of record. On the first boot after the upgrade you lose:

• In-flight webhook delivery retries. Postgres is the source of truth for deliveries, and a periodic poller re-attempts any pending row whose next attempt time has passed. They go out later, not never.
• Every active account lockout. An account mid lockout gets a clean slate, and the next bad attempt counts from zero.
• Current rate-limit windows. Counters reset to zero for a moment.
• In-flight PKCE state for OAuth and OIDC sign-ins. A sign-in caught mid-flow has to be started again.

Nothing in Postgres is affected. Compose never deletes a volume it no longer declares, so the old one just sits there. Once the deploy is healthy, remove it yourself:

docker volume rm rekey_redis   # or redis-data, for docker-compose.api.yml

Before you upgrade

2.2.0 is a minor release and it has breaking changes. Several migrations rewrite rows on tables that sign-in depends on, one builds five indexes that block writes while they build, and docker-compose.prod.yml now requires three proxy secrets it did not require in 2.1.x. Read the migration and rollback section of DEPLOY.md and the breaking-changes list in the changelog first. If you want the panel behaviour without the cache, set OPERATOR_AUTH_CACHE_TTL_MS=0: the prefetch and rate-limit fixes do not depend on it.

Where next

docs/rate-limits.md has every bucket, every ceiling and the arithmetic behind each number, plus how the API decides the client address behind a proxy. DEPLOY.md has the Valkey section and a table of what is cached for how long. If you have not used the panel much yet, the operator panel guide is the tour.