4

For the SaaS infrastructure I'm building, we need to send transactional emails and run other long-running tasks on a separate job server. I can communicate via a queue, such as AWS' SQS or via a database table.

But what exactly should I send from the web server to the worker server via the queue mechanism?

  • Should I send JSON text that has the entire email?
  • Or should I send a request to send an email called "foobar" with parameters a, b, and c?
  • Or something else?
2

In my opinion, it would be better if you could send the data in the form of strongly-typed business entity. On the worker server, you would have the benefit of manipulating the data with the help of intellisense in the IDE. You can easily write the code to formulate the email message with the help of business entity.

And try to send the request in the fire & forget mode and expect no response from the worker server. For instance, in the Microsoft WCF one-way communication data contracts can we used to address such needs.

If your message text size is small, then JSON would be good enough.

2

Choice B is superior. Enqueue a job by sending a Foobar(a, b, c) message to the long-running job queue. Do not pre-format the message in your web server code. Sending job parameters is better for both software engineering and practical site performance/reliability reasons.

TL;DR Explanation

There are two key reasons: First, modularity. Sending notification messages isn't a web serving function. Decoupling it is a cleaner, more modular design. A separate back-end process for handling Foobar and similar long-running jobs will enjoy the simplicity, maintainability, and other benefits that modularity and separation of concerns bring.

An even more urgent, practical, frequently-unappreciated motivation: You do not want to do any more work than necessary in your web serving code. Doing so can cause you major performance and site reliability problems.

It seems incredible in a multi-core, multi-threaded, multi-GB age that resources would be meaningfully constrained in your web server. Performance should not be a problem, and failure because of resource exhaustion? It seems laughable. But consider:

  • Most web sites now run on cloud resources. Hosting operators (whether Amazon, Heroku, Digital Ocean, or your own IT shop) use virtualization, throttling, and quotas to aggressively pack as many customers and workloads as possible onto each physical server. They similarly multiplex and control memory, storage, network bandwidth, and other resources. They have become exceedingly efficient at it.

    However powerful your server was when it left the factory, your workload gets only a sliver of it--and its key system resources like CPU cache and physical memory are already well-taxed handling other apps and workloads than yours. It may seem like you "own the place," but that's the virtualization illusion. Rarely is it so. Hosting operators aren't terribly generous with resources. Their pricing tiers are often directly tied to how many worker processes, CPU-minutes, GB RAM, GB storage, etc. you get. They have little incentive to be generous or flexible. So typical web-serving resources are much more tightly constrained than you might imagine.

  • Yet web servers are doing quasi real-time processing of network requests. They already have much to do, such as making requests on a DBMS, other shared middleware, storage, and remote servers, then filling out content templates to return to the user. They have to do that under tight latency constraints. If you have any kind of traffic volume whatsoever, each web-serving thread or process needs to complete many, many such requests per second. Especially long-running web workers (say, because they took extra time to access a slow API or to format an email message) can timeout. On some systems (e.g. Heroku), the timeout of one web worker kicks off a longish recovery process that, under load, can lead to other web requests being delayed and possibly timed out. Now you have the possibility of cascading failure.

    Even if you have light volume, web traffic is inherently burst-y. Have one of your links mentioned on a popular tweet or post and traffic soars instantaneously. Even "light duty" sites are on a much tighter timeline than you probably think, at least during burst moments.

    Let's say you can handle 50 concurrent requests. Maybe that seems "way more than enough!" because you generally get no more than 5 concurrent requests, "on average." Everything's golden, right? No. Even if your typical request count is 4, 2, 0, 10, 4, in a bursty moment you might get a few peak seconds of the day with a request sequence like: 21, 47, 37, 65, 31, 28. If you're doing more work than you have to on those requests, you're at risk on the 47, doubly at risk on the 47,37 one-two, and in mortal danger at 65.

    These are exactly the kind of burst patterns I pull out of logs. "But that can't be! Our site isn't that heavily loaded!" No, it's not. 86,381 seconds of each day, it's great. But those other 19 seconds, when by chance and happenstance you take >10x the average request load for a few seconds? You're at risk of a rolling disaster. If your web workers are efficient and complete their tasks quickly, queuing may save you. But if they run long and can't catch up, uh oh. You max out your worker pool (whether processes, threads, or event slots), start to generate timeouts and failures. Users and API callers get impatient and start requesting refreshes. In this process, the first failures/timeouts begin to cascade, causing subsequent ones. Suddenly your site is unresponsive, then essentially offline, then needs a hard reboot because it's exceeded some "maximum error threshold." Now your site's offline, not serving users or taking revenue. Ouch. You only a second or two of fumbling for the whole thing to topple. 19 over-peak seconds a day isn't a lot...but it's more than enough to see 5 or 10 failure clusters and possible downtime events per day, with a handful of them coming during your peak hours. A small number of over-max events can wreak havoc. And in practice, they do.

    Worse, this is a transient problem, often very hard to debug ("We have plenty of resources! It can't be that!"), not easily proven (the 19 bad seconds constitute just 0.02% of the day, have to be found among millions of log records, and are not clearly marked "hey, problem is here!!"), and are worst at peak times when management is most concerned about the site. Everyone's angry and frustrated with the downage. Yet your management--even if otherwise technically adept--may not understand that you need more system resources, since "the site is usually lightly loaded!" and 19 overages / 86,400 seconds per day doesn't seem like that much. Devoting more resources costs more. Sometimes 2x or 3x more, because they thought everything'd be great on the "small" tier, when what was needed for truly reliable operation was the "large" instance size--and two of them. Even if you can get their assent, with some hosts, it is flat-out impossible to buy more CPU or memory for either love or money.

    This is the nightmare scenario. Does it happen every time? No. But it does happen--and when it does, it's ugly. Younger me would have been skeptical that it could happen, or that adding a few more tasks to web serving would make any practical difference. Sadder but wiser me knows better. Having been called in as a DevOps engineer to fix sites that are slumping or falling down, often at the worst possible times and often for no clear reason, many of them turned out to be web worker exhaustion or timeouts leading to cascading failure. It can be Python and Django, Ruby on Rails, or PHP and WordPress. I've seen no language, framework, or concurrency architecture that's inherently immune. It's a composite failure that occurs when you combine long-running web requests, strongly virtualized and quota-limited web hosts, and the economics of tiered pricing--in other words, the modern outsourced cloud environment. It can be addressed, but it's painful to do so after the fact.

So if you expect any scale of web traffic at all, web workers should seen as quasi real-time request handlers. That means make them crisp and efficient. Not "throw anything at them" general-purpose jobs. Thus, they should emit short back-end requests, not pre-format them. It's also better modular design.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.