Best way to process a queue in C# (PDF treatment)

Question

First of all let me expose what I would like to do : I already dispose of a long-time running webapp developed in ASP.NET (C#) 2.0. In this app, users can upload standard PDF files (text+pics). The app is running in production on a Windows Server 2003 and has a dedicated database server (SQL server 2008) also running Windows Server 2003. I myself am a quite experienced web developer, but never actually programmed anything non-web (or at least nothing serious).

I plan on adding a functionality to the webapp for which I would need a jpg snapshot of each page of the PDF. Creating these "thumbnails" isn't the big deal as such, I already do it inside my webapp using ghostscript. I've only done it on 1 page documents for now though, and the new functionality will need to process bigger documents. In order for this process to be transparent aswell for the admins as the final users, I would like to implement some kind of queue to delay the processing of the thumbnails. There again, no problem to create the queue, it will consist of records in a table, with enough info to find the pdf document back.

Then I will need to process this queue, and that's were my interrogations start. Obviously the best solution to process it isn't an ASP script or so, so I will have to get out of my known environment. No problem, but I have no idea which direction to go. Therefore, a few questions :

• What should I develop ? I presumably need something that is "standby" on the server, runs when needed, then returns to idle state until further notice.Should I be looking into Windows service ? Is there another more appropriate type of project ?

• Depending on the first answer, what will be the approach ? Should I have somehow SQL server "tell" the program/service/... to process the queue, or should I have that program/service/... periodically check the state of the queue and treat new items. In both case, which functionality can I use ?

• we're not talking about hundreds of PDF a day (max 50 maybe), I can totally afford to treat the queue 1 item at a time. Can you confirm I don't have to look much further on threads and so ? (I found a lot of answers talking about threads in queue treatment, but it looks quite overkill for my needs)

• Maybe linked to the previous question : what about concurrent call to the program, whatever it is ? Let's suppose it is currently running, and a new record comes in the queue, what should be the behaviour ?

I don't need much detailed answers and would already be happy with answers like "You can do the processing with a service, and yes it's possible to have sqlserver on machine A trigger a service start on machine B" or "You have to develop xxx and then use the scheduler to run it every xxx minutes". I don't mind reading articles and so, but I can hardly afford to spend too much time learning stuff to finally realize I went the wrong way for this project, so basically I'm trying to narrow down the scope of matters I need to investigate.

Thanks for reading me, I hope I'll find some helping hands on here :-)

Answer 1

Your requirements are pretty mild, so you can afford to write a very simple / easy to maintain solution.

In your case I'd probably write a console app & launch it via the task scheduler. The app would read the "queue", process any existing requests, then exit. You could schedule it to run every 5 - 10 minutes or so. Updating the code is easy - overwrite the file between scheduled runs.

You could also use a Windows Service (using TopShelf for example) - they're pretty easy to write, but then instead of exiting when finished you'd need to sleep. Installing a new version gets to be a little bit more of a hassle than a console app, but not much.

If you start looking at higher volumes, you'd eventually want to eliminate the polling and probably move the queue out of the database & into something else (I have mixed feelings about MSMQ)

Answer 2

Using C# console apps which are run from the Windows Scheduler is a dirt simple way of doing this, and you have severale benefits in easy deployment, reduced problems from any memory leaks, besides you save a lot of energy from not reinventing yet another scheduler.

Answer 3

You've got a few different options -

1. Kick off a separate process/thread from your ASP.NET code to do the work. I would not recommend this approach, you'll have issues with ghost processes, IIS terminating things for you, user account security problems, etc etc.

2. A windows console application that you run periodically using the windows scheduler. Nice, simple, easy to maintain and deploy, easy to test. Easy for someone else to figure out what you've done - they can log onto your production box and see the scheduled task. The app checks the DB for what work needs to be done, does it, logs stuff to disk (or even better your DB, or maybe even the event log if you've got a good sysadmin) and then exists. I would go down this path for the moment. You can always stick your code into a DLL and "upgrade" it to the next option...

3. A windows service application. The nuclear bomb approach. Good if you need it to run more often like every minute or so. If you want to avoid DB polling, you could have it open up a WCF endpoint and talk to it directly. I have done this approach before and it works great, but it's probably overkill.