Servicing background tasks on a large site

Question

We're dealing with an interesting problem on StackOverflow.

We've got a whole bunch of little "needs to be done soon-ish" tasks. An example is updating "Related Questions" lists. What we've done in the past is to piggy-back those tasks onto some users' page loads.

This was never ideal, but it wasn't really noticeable. Now that SO has passed the 1,000,000 question mark, those unlucky users are starting to feel it.

The natural solution is to actually push these tasks into the background. There are two broad ways of doing this I'm considering.

1. In IIS as a custom Thread-Pool/Work-Queue

Basically, we spin up a few (non-ThreadPool, so as to not interfere with IIS) threads and have them services some collections we're shoving Funcs into.

The big pro here is simplicity. We don't have to worry about marshaling anything, nor do we have to make sure some external service is up and responding.

We also get access to all of our common code.

The con is, well, that we shouldn't use background threads. The objections I know of are all centered around starving IIS (if you use ThreadPool) and the threads dieing randomly (due to AppPool recycling).

We've got existing infrastructure to make the random thread death a non-issue (its possible to detect a task has been abandoned, basically), and limiting the number of threads (and using non-ThreadPool threads) isn't difficult either.

Am I missing any other objections to in IIS process thread-pooling/work-queues?

Moved to StackOverflow, as it wasn't really addressed here.

2. As a Service

Either some third-party solution, or a custom one.

Basically, we'd marshal a task across the process boundary to some service and just forget about it. Presumably we're linking some code in, or restricted to raw SQL + a connection string.

The pro is that its the "right way" to do this.

The cons are that we're either very restricted in what we can do, or we're going to have to work out some system for keeping this service in sync with our code base. We'll also need to hook all of our monitoring and error logging up somehow, which we get for free with the "In IIS" option.

Are there any other benefits or problems with the service approach?

In a nutshell, are there unforseen and insurmountable problems that make approach #1 unworkable and if so are there any good third-party services we should look into for approach #2?

Answer 1

A few weeks back I asked a similar question on SO. In a nut shell, my approach for some time now has been to develop a Windows Service. I would use NServiceBus (essentially MSMQ under the covers) to marshal requests from my web app to my service. I used to use WCF but getting a distributed transaction to work correctly over WCF always seemed like a pain in the ass. NServiceBus did the trick, I could commit data and create tasks in a transaction and not worry whether my service was up and running at the time. As a simple example, if ever I needed to send an email (for example a registration email) I would create the user account and fire off a signal to my Windows Service (to send the email) in a transaction. The message handler on the service side would pick up the message and process accordingly.

Since ASP .NET 4.0 and AppFabric have been released, there are a number of viable alternatives to the mechanism above. Referring back to the question I mentioned above, we now have AppFabric's AppInitialize(via net.pipe) as well as ASP .NET 4.0's Auto-Start feature which make developing Windows Services as web apps a viable alternative. I have started doing this now for a number of reasons (biggest one being deployment is no longer a pain in the ass):

1. You can develop a web UI over your service (since it's running as a web app). This is extremely useful to see what is happening at runtime.
2. Your deployment model for your web apps will work for your service application.
3. IIS provides a few neat features for handling application failures (similar in some respects to a Windows Service).
4. Web developers are very familiar with developing web apps (naturally), most don't know much about best practice when developing a Windows Service.
5. It provides a number of alternatives to exposing an API for other apps to consume.

If you go this route (forgive me for copying and pasting from my original post) I would definitely consider running the background logic in a separate web application. There are number of reasons for this:

1. Security. There may be a different security model for the UI displaying information about the running background processes. I would not want to expose this UI to anyone else but the ops team. Also, the web application may run as a different user which has an elevated set of permissions.
2. Maintenance. It would be great to be able to deploy changes to the application hosting the background processes without impacting on user's using the front end website.
3. Performance. Having the application separated from the main site processing user requests means that background threads will not diminish IIS's capability to handle the incoming request queue. Furthermore, the application processing the background tasks could be deployed to a separate server if required.

Doing this gets back to the marshaling aspect. WCF, NServiceBus/RabbitMQ/ActiveMQ etc., vanilla MSMQ, RESTful API (think MVC) are all options. If you are using Windows Workflow 4.0 you could expose a host endpoint which your web app could consume.

The web hosting approach for services is still fairly new to me, only time will tell if it was the correct choice. So far so good though. By the way, if you don't want to use AppFabric (I couldn't because for some bizarre reason, Windows Server Web Edition ain't supported), the Auto-Start capability mentioned in the Gu's post works nicely. Stay away from the applicationhost.config file though, everything in that post is possible to setup through the IIS console (Configuration Editor on the main server level).

Note: I had originally posted a few more links in this message but alas, this is my first post to this exchange and only one link is supported! There was basically two others, to get them Google "Death to Windows Services...Long Live AppFabric!" and "auto-start-asp-net-applications". Sorry about that.

Answer 2

There is actually a third way in Windows to run background services, and it is very common in the UNIX world. The third way is a CRON job that runs a piece of your infrastructure. In Windows this is known as the task scheduler and is very common for running code on a scheduled basis. To use this you would create a command-line app that is executed on a pre defined schedule. The advantage of this is that you don't have to worry if the process stays up and running like a service, because if it fails for some reason, it will just start up next time.

As for marshaling specific tasks, you really just need to store these tasks in a persistent binary storage. Until the command line app picks them out of the storage and executes them. I have done this in the past using the Cassandra database as a Session State Provider for stuffing background tasks for specific users in the Cassandra database, and then having the commandline pick them out and execute them for the user.

This may not have been the typical marshaling solution, but it worked out very well for me and it turned out to be a very elegant solution, because the scheduled tasks survived shutdowns, network problems, and any machine could execute the task since it was centrally stored.

Shameless promotion, but this is my project and the solution I just briefly detailed is why I created the project: http://github.com/managedfusion/fluentcassandra/

Answer 3

Cron + Web App

This is a battle-tested design that scales horizontally along with your web farm and ensures that you're using the web technology stack you already know.

Here's how it works:

1. Create a controller/action in your web application to handle scheduled background tasks. By convention, I usually call mine http://mydomain.com/system/cron.
2. For security, this action should be locked down to only authenticated IP addresses on the local network.
3. On a seperate machine, install Wget and setup a Scheduled Task to have wget fetch the resource from step 1. You can make the task run as frequently as you want (I usually opt for 30 seconds). Don't forget to pass the appropriate cookie argument to Wget so that it authenticates to your web app.
4. For redundancy, you can also install a second scheduled wget on a second machine.

Hooray! Now you have a route that will be called every 30 seconds. And if the request takes 5 minutes to process, nobody will care, because it's not part of a user's page request.

The cron action ends up looking very simple: he has a list of methods to execute on a certain frequency. When a request comes in, he sees if there is a method that needs to be executed and calls the appropriate method. This means that you can control the schedule in your database, where you probably already have lots of other important configuration data for your site.

More importantly (for you), this means that your jobs don't have to be called on a fixed schedule. You can write any logic you want for determining when to execute a method.

Pros and Cons

Pros

• You're already very good at writing ASP.NET MVC code, so this lets you write your background tasks in the same platform that you write the rest of your solution in.
• The tasks run in the same context as your web app, so you can share the cache and make use of helper methods that already exist.
• If you have wget fetch a load-balanced URI, then your background tasks are now load-balanced as well.
• Simultaneous deployment - you don't have to worry about syncing your web app with your background task logic, because they're all in the same deployment.

Cons

• Over the years, a few people have told me this design is "highly coupled", but when pressed they haven't been able to articulate why that is a bad thing.

Note: If there are any questions or concerns, please add a comment. I'm happy to elaborate.

Answer 4

I have tried and used just about every possible way of doing this in my current application. I started out doing the same thing that you currently do, piggy back on a user request to fill the data and then cache it going forward. I realized this was a bad idea as well (especially as you scale to multiple web servers, more users take the hit).

I also have had a scheduled job that hits a URL in the ASP.NET app - this is a decent solution but it starts to break down the minute you scale past 1 web server.

Currently I use two different methods, both using Quartz.NET which is a great little library. The first is Quartz.NET running in-process with ASP.NET, it is setup in the global.asax and runs every couple of minutes. I use this to update the ASP.NET cache out of band which is the only reason it is run as part of ASP.NET.

The second is that I wrote a library to wrap Quartz.NET called DaemonMaster - it makes it easy to drop a DLL into a directory and have it run in a Windows service. I found it helps avoid some of the annoying parts of working with a Windows Service and also cleans up the Quartz.NET api some. The services that run through DaemonMaster are of two different flavors, the first are jobs that need to run every night or every X minuts. The other jobs work off of a queue based on data coming in from the ASP.NET application. The ASP.NET app drops JSON objects on RabbitMQ and the services poll RabbitMQ then process the data.

Based on this I would suggest you go with a Windows service (and check out DaemonMaster) and if needed use a queue like RabbitMQ for passing the data from the ASP.NET app to the services - it has worked the best out of all these solutions. If you are loading cache then running in ASP.NET makes sense, otherwise I don't think it does.

• DaemonMaster: http://github.com/adzerk/DaemonMaster
• Quartz.NET: http://quartznet.sourceforge.net/

Answer 5

I'd do it the right way and have a Windows Service running that monitors a "queue". I say "queue" because programming w/ MSMQ is akin to sticking hot pokers in your eyeballs.

I've fallen in love with the simplicity of Delayed::Job in Rails, and something similar could easily be done in .NET.

Basically you add any sort of SomethingOperation (something that has an Perform() method). Then just serialize the relevant parameters, give it a priority, some sort of default retry behavior and stuff it into a database.

Your service would just monitor this and work the jobs in the queue.

Answer 6

We've been pretty happy with a Service Bus / Message Queue / Service approach. The basic architecture is this.

Website sends message to queue

bus.Send(new ProjectApproved()); // returns immediately

Windows service receives and processes message in its own time

public class DoesSomethingAwesome : ConsumerOf<ProjectApproved>
{
   public void Consume(ProjectApproved Message)
   {
      // Do something "offline"
   }
}

The advantage is that there is no delay for the front-end service that users are connected too. The windows service can be shutdown and be upgraded without interruption to the main site. Plus it's extremely fast.

If you cannot store all of your data within the message you can always store it and retrieve it later. I suggest using a document storage mechanism such as: RavenDB or MongoDB where it's very straight forward to store your classes without changes.

Website sends message to queue

// Save your object
store.Save(completeProject);

// Send a message indicating its ready to be processed
bus.Send(new ProjectApproved() { ProjectId = completeProject.Id });

Windows service receives and processes message in its own time

public class DoesSomethingAwesome : ConsumerOf<ProjectApproved>
{
   public void Consume(ProjectApproved Message)
   {
      // Retrieve your object back
      var completeProject = store.Get(Message.ProjectId);
   }
}

To make things simple we use: Rhino ESB and Topshelf. The configuration is extremely simple and putting this in place for an existing application has proved to take very little time.

Answer 7

I'm curious why a combination of the two isn't a viable option. Right now you trigger jobs on page views, with some unlucky sap getting stuck waiting 10 seconds for the page to come up. At least that's my understanding of your current method.

However those jobs are taking longer and longer to run as the site grows, and you don't want to derail the user experience on the site. Not even for a few (Or maybe a lot) unlucky users through out the day, so now you're thinking about scheduling jobs in the background.

I don't see why a background job run at regular intervals can't mimic a visitor. Now I'm not a Windows programmer, but in the Linux world I would set up a cron job that runs at a regular interval, and it would have 2 lines of code.

#!/bin/bash
wget -O /dev/null http://stackoverflow.com/specially_crafted_url

It combines the pros of both systems. It's done in the background. It doesn't effect users. It still uses a page view to kick off the job. I've seen this approach used before. It tends to be the middle ground between the simple ways of old, and the more complex ways coming down the road.

Update

I think you can get around the load balancing issue by running the job runners on the web servers themselves. The job runner pulls a URL out of the job queue, and runs it like so:

wget -O /dev/null http://localhost/specially_crafted_url

Because of the nature of job/messaging queues, the jobs will get evenly distributed among the job runners, which means the specially_crafted_url is eventually distributed among your web servers.

Answer 8

I think the con with the pure service approach is that you have code scattered into the service and away from the core app.

Here's what we've done with large background non time-sensitive jobs, which keeps the code together and simplifies the service:

1. Create a jobs queue (either in-memory or DB, whatever persistence is needed for the types of jobs)
2. Create a web service that will execute the queued jobs
3. Dead simple service app that calls the web service at a specified interval, leave all the complex stuff (job retrieval and execution) to the web service in your core codebase.

Even simpler, just make the call in a console app and use Task Scheduler or VisualCron to turn it into a "service".

Answer 9

I liked TopShelf. Keeps the simplicity, yet still do it the proper way running as a Windows Service. Basically create a Console App, add about 15-20 lines of code, then it installs as a service.

http://code.google.com/p/topshelf/

Answer 10

How about having a very simple Windows service that runs on the web server and periodically hits a maintenance URL which does your miscellaneous tasks. Have it throttle how much work it does in any given request.

Answer 11

I'm going to buck the apparent trend here and suggest going for the in-IIS model. I've used it myself and it works really well. It's really not that hard to implement a decent thread-pool class (over the years, I've extended my thread pool class to support dynamic creation and destruction of threads, retrying of jobs and so on). Advantages are:

• No external service to monitor
• Simplicity of implementation: no cross-process marshalling, no advanced job monitoring
• You're still inside your IIS process, so you can do all of your usual logging and so on (no need for multiple log files)
• Vastly simplified deployment (when you update a service, you have to stop the service, copy the files, start the service - this is in addition to your usual updates to the website code)

In my opinion, an in-IIS solution is simply the "next step up" from piggybacking the work onto random page views.

Answer 12

Resque is nice. Or even Kthxbye if you need to be notified of the resulting value once it's completed.

Both Redis/Ruby based tho.

Honestly, if you're doing a service-based approach, it really doesn't need to be super-integrated with your current platform, which I feel is a plus. I would hope it could be a set-and-forget system that would run (with monitoring of some kind) and complete jobs. I'm not sure it has to be run on the same platform at all since its just updating/modifying database info.

Pretty sure you could get away with a lot more for a lot less if you farmed this kinda work out to a separate entity, especially since it seems you're dealing with threading issues. Both Resque and Kthxbye move the processing out to separate processes to allow the OS to handle the concurrency.

Resque

Kthxbye

Answer 13

I would use a WAS hosted WCF service listening to an MSMQ Queue.

Pro's

• Fire and forget one way messages from the web app

• MSMQ/WCF throttling and retry

• Guaranteed delivery ;D

• Dead Letter management

• Distributed processing

• WAS / MSMQ activation

Con's

• MSMQ (it's not dead ... Yet)

The MSMQ features in WCF makes using MSMQ really nice. Yes you will bleed on the configuration but the benefits will outweigh the sacrifice.

Answer 14

I've run into this a couple of times when developing web applications. We've been solving it by creating a windows console application that carries out the task, and creating a scheduled task that runs every so often to actually do the task.

Answer 15

You can shunt work onto a background thread ( or many background threads ) using Rx and something like the following:

var scheduler = new EventLoopScheduler( SchedulerThreadName );
_workToDo = new Subject<Action>();
var queueSubscription = _workToDo.ObserveOn( scheduler ).Subscribe( work => work() );
_cleanup = new CompositeDisposable( queueSubscription, scheduler );

To use:

var work = () => { ... };
_workToDo.OnNext( work ); // Can also put on error / on complete in here

Host all that inside a class that there is only ever one of ( aka a singleton, but do it properly - use you IoC container to determine the lifestyle ).

You can control the size of the thread pool etc by writing a custom scheduler in place of using the EventLoopScheduler ( which runs a single thread ).

Answer 16

I have implemented this type of thing a few times. On windows, I set up a python command-line program that does something at various times. This program also exposes an xmlrpc interface on a port. Then, a scheduled-task job runs every minute and queries the xmlrpc interfaces. If they are not up, it tries to launch them. If it can't, it emails me.

The advantage is that the job that runs isn't cron or schedule bound. I have a process job that runs every seconds, but will wait longer and longer between starting a new job depending on whether it had work to do. Also, it can be used to act intelligently based on the result. Got a 500 error? Got a really long delay? Do something else. Notify another service. Etc.

And the same system works on unix, with minor modifications.

Answer 17

I don't have an answer for you myself, but the problem rung a bell - I remember some random guys discussing it on a podcast once.

Spolsky: I noticed one of the questions you asked on the blog was how should you handle maintenance recurring tasks in general?

Atwood: Yes.

Spolsky: Is that a fair characterization? Every website has some tasks that you don't want to execute at the time a web page is loading, but you want to execute with some recurrence of some sort.

Atwood: Ya, background tasks sort of thing.

Spolsky: Ya, so what did you figure out?

Atwood: Well, I originally asked on Twitter, because I just wanted something light weight. I really didn't want to like write a windows service. I felt like that was out of band code. Plus the code that actually does the work is a web page in fact, because to me that is a logical unit of work on a website is a web page. So, it really is like we are calling back into the web site, it's just like another request in the website, so I viewed it as something that should stay inline, and the little approach that we came up that was recommended to me on Twitter was to essentially to add something to the application cache with a fixed expiration, then you have a call back so when that expires it calls a certain function which does the work then you add it back in to the cache with the same expiration. So, it's a little bit, maybe "ghetto" is the right word.

Answer 18

Task Queue Java API Overview

Task Concepts
In App Engine background processing, a task is a complete description of a small unit of work. This description consists of two parts:

• A data payload which parameterizes the task.
• Code which implements the task.

Tasks as Offline Web Hooks
Fortunately, the Internet provides such a solution already, in the form of an HTTP request and its response. The data payload is the content of the HTTP request, such as web form variables, XML, JSON, or encoded binary data. The code reference is the URL itself; the actual code is whatever logic the server executes in preparing the response.

Answer 19

Do Both

Add a optional parameter to the question path that does the work you are currently piggybacking on user requests:

Servicing background tasks on a large site

Create a console app that runs on each server and opens the IIS log shared binary and reads it to the current end of the file. Use a filesystemwatcher or a timed interval to read forward to collect updates as IIS flushed the log.

Use this information to determine what pages have been currently viewed.

Use the page urls from the parsed log to call the "extrastuff" version of the url on localhost with a webclient object.

Add in some code to switch files at the end of each log period or restart the process each log period.