Most web sites now run on cloud resources. Hosting operators (whether Amazon,
Heroku, Digital Ocean, or your own IT shop) use virtualization, throttling,
and quotas to aggressively pack as many customers and workloads as possible
onto each physical server. They similarly multiplex and control memory,
storage, network bandwidth, and other resources. They have become exceedingly
efficient at it.
However powerful your server was when it left the factory, your workload gets
only a sliver of it--and its key system resources like CPU cache and physical
memory are already well-taxed handling other apps and workloads than yours. It
may seem like you "own the place," but that's the virtualization illusion.
Rarely is it so. Hosting operators aren't terribly generous with resources.
Their pricing tiers are often directly tied to how many worker processes,
CPU-minutes, GB RAM, GB storage, etc. you get. They have little incentive to
be generous or flexible. So typical web-serving resources are much more
tightly constrained than you might imagine.
Yet web servers are doing quasi real-time processing of network requests. They
already have much to do, such as making requests on a DBMS, other shared
middleware, storage, and remote servers, then filling out content templates to
return to the user. They have to do that under tight latency constraints. If
you have any kind of traffic volume whatsoever, each web-serving thread or
process needs to complete many, many such requests per second. Especially
long-running web workers (say, because they took extra time to access a slow
API or to format an email message) can timeout. On some systems (e.g. Heroku),
the timeout of one web worker kicks off a longish recovery process that, under
load, can lead to other web requests being delayed and possibly timed out. Now
you have the possibility of cascading failure.
Even if you have light volume, web traffic is inherently burst-y. Have one of
your links mentioned on a popular tweet or post and traffic soars
instantaneously. Even "light duty" sites are on a much tighter timeline than
you probably think, at least during burst moments.
Let's say you can handle 50 concurrent requests. Maybe that seems "way more
than enough!" because you generally get no more than 5 concurrent requests,
"on average." Everything's golden, right? No. Even if your typical request
4, 2, 0, 10, 4, in a bursty moment you might get a few peak seconds
of the day with a request sequence like:
21, 47, 37, 65, 31, 28. If you're
doing more work than you have to on those requests, you're at risk on the 47,
doubly at risk on the 47,37 one-two, and in mortal danger at 65.
These are exactly the kind of burst patterns I pull out of logs. "But that
can't be! Our site isn't that heavily loaded!" No, it's not. 86,381 seconds of
each day, it's great. But those other 19 seconds, when by chance and
happenstance you take >10x the average request load for a few seconds? You're at risk of a rolling
disaster. If your web workers are efficient and complete their tasks quickly,
queuing may save you. But if they run long and can't catch up, uh oh.
You max out your worker pool (whether processes, threads, or event
slots), start to generate timeouts and failures. Users and API callers get
impatient and start requesting refreshes. In this process, the first
failures/timeouts begin to cascade, causing subsequent ones. Suddenly your
site is unresponsive, then essentially offline, then needs a hard reboot
because it's exceeded some "maximum error threshold." Now your site's offline,
not serving users or taking revenue. Ouch. You only a second or two of
fumbling for the whole thing to topple. 19 over-peak seconds a day isn't a
lot...but it's more than enough to see 5 or 10 failure clusters and possible
downtime events per day, with a handful of them coming during your peak hours.
A small number of over-max events can wreak havoc. And in practice, they do.
Worse, this is a transient problem, often very hard to debug ("We have plenty
of resources! It can't be that!"), not easily proven (the 19 bad seconds
constitute just 0.02% of the day, have to be found among millions of log
records, and are not clearly marked "hey, problem is here!!"),
and are worst at peak times
when management is most concerned about the site. Everyone's angry and
frustrated with the downage. Yet your management--even if otherwise
technically adept--may not understand that you need more system resources,
since "the site is usually lightly loaded!" and 19 overages / 86,400 seconds
per day doesn't seem like that much. Devoting more resources costs more.
Sometimes 2x or 3x more, because they thought everything'd be great on the
"small" tier, when what was needed for truly reliable operation was the
"large" instance size--and two of them. Even if you can get their assent, with
some hosts, it is flat-out impossible to buy more CPU or memory for either
love or money.
This is the nightmare scenario. Does it happen every time? No. But it does
happen--and when it does, it's ugly. Younger me would have been skeptical
that it could happen, or that adding a few more tasks to web serving would
make any practical difference. Sadder but wiser me knows better. Having been
called in as a DevOps engineer to fix sites that are slumping or falling down,
often at the worst possible times and often for no clear reason, many of them
turned out to be web worker exhaustion or timeouts leading to cascading
failure. It can be Python and Django, Ruby on Rails, or PHP and WordPress.
I've seen no language, framework, or concurrency architecture that's
inherently immune. It's a composite failure that occurs when you combine
long-running web requests, strongly virtualized and quota-limited web hosts,
and the economics of tiered pricing--in other words, the modern outsourced
cloud environment. It can be addressed, but it's painful to do so after the