Launch offer: the first 5 clients receive a free 2-hour virtual consultation.·3 client spots open this quarter.
twill/softwareSoftware engineering studio
← All posts
· 5 min read

2.9s to 300ms: A Systematic Approach to API Latency

PerformanceEngineeringAPIs

A slow API rarely has one dramatic cause. It has a dozen small ones, each invisible on its own, that add up to a request taking 2.9 seconds when it should take 300 milliseconds. Teams that fix this do not look for the cleverest optimization. They measure first and fix in a sensible order.

I cut the p95 of a high-traffic catalog API from 2.9 seconds to roughly 300 milliseconds this way, with no rewrite. The method below is what that work looked like.

Measure before you touch anything

The most common mistake in performance work is optimizing the thing you assume is slow. It is almost never that thing.

Start with the distribution. A p50 of 400ms with a p95 of 2.9s is a different problem from a flat 2.9s on every request. The first is tail latency: a subset of requests hitting a slow path. The second is systemic: everything is slow. They get fixed differently, and an average hides the difference.

Then find where the time goes. Wrap the major phases of a request in timing or tracing: authentication, database calls, downstream services, serialization. Pull up one genuinely slow request and read it end to end. You want the largest contiguous block of time. That block is your first target, and it is often somewhere nobody suspected.

The usual suspects

Once you can see the breakdown, the cause is usually one of a short list.

N+1 queries. The endpoint runs one query for a list, then one more query per item. Fifty items means fifty-one round trips. Each query is fast, so nothing stands out in a profile of any single call.

-- N+1: one query per product
SELECT * FROM prices WHERE product_id = $1;   -- repeated 50 times

-- One round trip
SELECT * FROM prices WHERE product_id = ANY($1::bigint[]);

Serial I/O that could run in parallel. Three independent downstream calls made one after another cost the sum of their latencies. Made together, they cost the slowest one.

// Sequential: ~ a + b + c
const inventory = await getInventory(id);
const pricing = await getPricing(id);
const reviews = await getReviews(id);

// Concurrent: ~ max(a, b, c)
const [inventory, pricing, reviews] = await Promise.all([
  getInventory(id), getPricing(id), getReviews(id),
]);

Cold or missing caches. Data that changes rarely gets recomputed or refetched on every request.

Oversized payloads and unindexed queries. Fields no client reads, lists with no pagination, and queries that scan a million rows to return ten.

These causes are mostly structural. You are rarely speeding up a hot loop. You are removing round trips and work that did not need to happen.

Fix in order of latency saved per hour of work

Rank the fixes by how much time each saves for the effort it takes, ignoring how interesting each one is.

  1. Remove the N+1. Batch the per-item queries into one. This is often the biggest win for the least work.
  2. Parallelize independent calls. Three serial waits collapse into one.
  3. Cache the stable data. A short-lived cache in front of slow-changing data is cheap and effective, but it brings an invalidation problem. Do it after the structural fixes so you know the cache is hiding nothing.
  4. Trim the payload and add the missing index. Return what the client uses, and check every query the endpoint runs with EXPLAIN.

Make one change at a time and measure after each. If you bundle three changes, you cannot tell which one helped, and you will not notice if one of them made something else worse.

Verify without breaking anything

A faster endpoint that returns different data is a bug. Batching and caching are the two easiest places to change behavior by accident. Before and after each change, compare responses for a set of real requests and confirm they match exactly.

Also test under realistic concurrency. A single warm request on a laptop tells you little about a database with fifty connections fighting over it. Use a load test that resembles production traffic, run it before and after, and compare the p95. The p50 usually looked fine all along, so it will not tell you whether you solved the problem you started with.

Watch the tail after you ship

Percentiles drift. A cache that works today can start missing when a new product category launches, and an index that fit in memory last quarter may not fit this quarter. Put the endpoint’s p95 on a dashboard with an alert a little above the new baseline, so a regression shows up as a graph moving instead of a customer complaint.

Know when to stop

At some point the next 50ms costs more engineering than it is worth. When the endpoint sits comfortably inside its latency budget and the remaining time is spread thinly across many small things instead of concentrated in one fixable block, stop. Performance work has steeply diminishing returns. The skill is recognizing when you have captured the easy 90% and moving on to something that matters more.

Nothing in this was a heroic rewrite. It was honest measurement, the largest block first, and a check after every step. Most slow APIs have the same handful of problems waiting to be found.

If your API is slow and you are not sure where the time goes, that is the kind of problem Twill Software diagnoses and fixes. The first consultation is free: get in touch.