Nginx — Remember to cache content until the cache is actually updated

Caching when stale

If you cache something for say 6 seconds in Nginx, then only one request comes through per 6 seconds right? Not necessarily. It depends on how you have configured the behavior for when a cached entry goes stale.

If the backend takes a number of milliseconds to update the cache, other requests that are let through during that time (let through since the cache is stale) could overwhelm the backend application server.

In the default example in the nginx blog they do not cater for this "slip-through":

proxy_cache_use_stale error timeout http_500 http_502 http_503 http_504;

However it can be mitigated. For all versions of nginx there is an updating value you can add to the proxy_cache_use_stale directive. It will continue serving cached requests even when the cache is stale

proxy_cache_use_stale updating […]

Here is a comparison between not having the updating parameter switched on and having it on, for a slow backend application server with a low tolerance for load. In both cases 10 simultaneous user agents each doing 1 request per second for 30 seconds. Framework used is Artillery.js. Check the 95th and 99th percentiles

Request latency:
min: 33.7
max: 16763.4
median: 37.9
p95: 7666.3
p99: 15260.1
Scenario counts:
0: 300 (100%)
Codes:
200: 300

Now with the updating parameter added:

Request latency:
min: 34.3
max: 726.9
median: 38
p95: 43.4
p99: 70
Scenario counts:
0: 300 (100%)
Codes:
200: 300