Search latency degradation after api v2.31.0

Date2026-09-11
SeveritySEV-2
Duration23 min (14:06–14:29 UTC)
Detected byLatency SLO pager
Incident leadMaya

Customer impact. Search was slow or timed out for roughly 8% of active sessions during the 23-minute window; p95 latency on /search rose from 180 ms to 2.4 s. No data was lost and no other endpoint was affected. The fix was a rollback; the root cause is understood and the permanent fix is scheduled below.

Timeline all times UTC

  1. 14:02Deploy of api v2.31.0 completes across all regions. Canary passed (error rate only).
  2. 14:06p95 on /search climbs from 180 ms to 2.4 s within two minutes. 
    14:06:12 WARN  search-api  p95=1840ms  window=60s  replica=search-ro-2
14:06:42 WARN  search-api  p95=2410ms  window=60s  replica=search-ro-1
14:07:05 ERROR search-api  upstream timeout after 2000ms  route=/search  n=41
  3. 14:09Pager fires for the search latency SLO (5-minute burn rate).
  4. 14:12Maya acknowledges, opens the incident channel, pulls in Tomas.
  5. 6 minutes: dashboards showed CPU flat, connections normal. Nothing obvious.
  6. 14:18Suspect identified. Release notes for v2.31.0 include a migration that sets jit = on for the search read replicas. Query plans on search-ro-1 confirm JIT compilation on the hot query. 
    search-ro-1=# EXPLAIN (ANALYZE) SELECT ... FROM documents WHERE tsv @@ $1 ...
 JIT:
   Functions: 14
   Options: Inlining true, Optimization true, Expressions true, Deforming true
   Timing: Generation 2.1 ms, Inlining 38.4 ms, Optimization 214.9 ms, Emission 96.2 ms, Total 351.6 ms
  7. 14:25Rollback to v2.30.4 started. Migration reverted on both replicas.
  8. 14:29Recovered. p95 back to 190 ms. Pager clears at 14:34 after the burn-rate window.

Root cause

The v2.31.0 migration enabled PostgreSQL's JIT compiler on the search read replicas, intending to speed up analytical queries on the same hosts. The planner also applied JIT to the search hot path: a full-text query that runs about 400 times per second. JIT compilation costs roughly 350 ms per plan, and because the query's parameters vary, plans were not being reused. Every search paid the compilation cost.

The canary did not catch it because the canary gate checks error rate, not latency, and the requests were succeeding, just slowly, until the upstream timeout started tripping.

What worked and what didn't

What worked

  • SLO-based paging fired three minutes after onset, without a manual threshold.
  • Rollback was a single command and took four minutes.
  • Release notes listed the config change, which is what led to the diagnosis.

What didn't

  • Six minutes were spent on host metrics before anyone read the release notes.
  • The canary gate is blind to latency regressions.
  • Database config changes ride along in app migrations, so they get app-level review, not DBA review.

Action items

ActionOwnerDue
Pin jit = off in the read-replica config; move the setting out of app migrations.MayaFri 2026-09-18
Add a p95 latency check to the deploy canary gate, per route.TomasNext sprint
Write down the rollback procedure used here as a runbook.PriyaThis week
Add "read the release notes first" to the incident-lead checklist.MayaThis week
Appendix: why JIT hurt here and not on the analytics replicas

The planner enables JIT when the estimated query cost exceeds jit_above_cost (default 100 000). The search query's cost estimate sits just above that line because of the GIN index scan estimate, so it qualifies, but it runs in under 5 ms without JIT. Analytical queries run for seconds and amortise the compile. The right fix is per-role or per-replica, not global.