Cache hit rate:
what it is and how it changes response time

Hit rate and miss rate defined

A cache request is either a hit, answered from the stored copy, or a miss, which has to go to the slower origin such as a database, disk or upstream server. Hit rate is the fraction of requests that are hits, and miss rate is the remainder.

Count the same thing in both terms. CDNs sometimes report byte hit rate (share of bytes served from cache) alongside request hit rate, and the two can differ a lot when large files behave unlike small ones.

What hit rate does to average latency

Take a cache that answers in 2 ms and an origin that takes 120 ms. The average over a mix of requests moves as follows.

Going from 80 to 92 percent more than halves latency, and the last stretch from 96 to 99 percent halves it again. The gain looks small in percentage points but is large because the origin cost is spread over a shrinking number of requests.

Misses decide how hard the origin works

The better lens is the miss rate, since that is the traffic the database or backend still has to handle. Moving from 92 to 96 percent hits cuts the miss rate from 8 to 4 percent, which halves origin load. The same two percentage points near 50 percent would make little difference to it.

This is why teams chase the tail. A backend sized for 80 requests per second has far more room than one sized for 200, and a surge in traffic is absorbed by the cache, not the database.

Hit rates compound across layers

Most requests pass through more than one cache: the browser, a CDN, then an application cache in front of the database. What reaches the origin is the product of the miss rates at each layer, so modest hit rates multiply into a large reduction.

Remember that each layer has its own definition and its own reporting. A CDN's hit ratio only covers the traffic it sees, so the overall figure has to be built from the layers, not read from one dashboard.

Common reasons for a low hit rate

• Cache keys that include noise such as tracking parameters, timestamps or user IDs, so identical content gets many keys.
• Time-to-live set too short for content that rarely changes.
• Cache too small, so useful entries are evicted before they are reused.
• Personalised or authenticated responses marked as uncacheable when parts of them could be shared.
• Very long-tail traffic where most items are requested once.

Fixing key normalisation usually gives the biggest jump for the least effort. Check what makes up the key before you buy more memory.

When the number can mislead

A high hit rate on trivial requests can hide a poor result on expensive ones. If 95 percent of hits are for tiny static files while the costly API responses almost always miss, the origin is still struggling.

Segment the metric by route, content type or customer tier. Also note what counts as a miss: stale-while-revalidate and conditional requests that get a 304 are handled differently by different systems, so confirm the definition before you compare tools or set targets.

Finally, the best hit rate is not always 100 percent. Content that must be fresh, such as a balance or inventory count, may deliberately bypass the cache.

Matching symptoms to causes

When the rate disappoints, the pattern tells you where to look before you change any settings.

Eviction policy matters too. Least-recently-used suits most workloads, but a one-time scan of the whole catalogue can push out useful entries, which is why some systems protect a hot set or admit new items more selectively.

The cost view and what to alert on

Hit rate also translates into money. Suppose a site serves 10 TB a month. With a byte hit rate of 92 percent, 0.8 TB is pulled from the origin; at 96 percent, only 0.4 TB. If egress or compute behind the origin is charged by volume, the second figure halves that bill.

For operations, a steady number matters more than a perfect one. Alert on a sudden fall relative to the usual level for that hour, since a drop from 94 to 85 percent usually signals a bad deploy, a changed key or a purge, long before users complain. Track the metric per route or content class so that a healthy aggregate cannot hide one struggling endpoint.

• Record hits, misses and bypasses separately; a bypass is not a miss in the same sense.
• Compare against the same weekday and hour, not just the previous minute.
• Pair hit rate with origin latency and error rate on the same dashboard.

One last caution concerns freshness. Raising the lifetime of cached entries is the cheapest way to improve the hit rate, but it also lengthens the time that users can see out-of-date content. Decide per content type how stale is acceptable, use explicit invalidation for the few items that must change at once, and treat the hit rate as one half of a pair, alongside a measure of how often users actually received something older than they should have.