1

I'm currently building an Asp.Net website that will allow to import data in several formats into a database. The import status of the the single import jobs are visible to all users and given that a user has sufficient privileges he can also stop an import started by another user. If an import goes through the data is written to a sql-server instance. (Data management ist done through nhibernate)

I would need some pointers on the best practices to construct a shared threadpool that allows cancellation of tasks.

Up to now i've thought that something like this could be implemented best using a blocking queue following a producer-consumer pattern. Each user can generate jobs that are gathered in a single service which then adds the job to the queue. The service would then also hold a separate list with the job id's and their corresponding cancellation tokens.

Edit: The version used is Asp.Net MVC 5. The blocking queue would utilize the Task Parallel Library. The size of the files to import ranges from 50kb to 5 mb.

4
  • Why do you think you need raw threads? What version of the framework are you working with? If it's anything remotely modern, then you'd be far better off using Tasks and letting the framework handle the threading.
    – RubberDuck
    Commented Jun 13, 2017 at 10:51
  • 1
    Oh sry. I meant tasks. I'll edit it immediately. The blocking queue that i'd implement would be using the TPL.
    – narain
    Commented Jun 13, 2017 at 10:56
  • How big are the files you are considering?
    – T. Sar
    Commented Jun 13, 2017 at 10:56
  • 50kb to 5mb. But due to the format the parsing can be quite time consuming.
    – narain
    Commented Jun 13, 2017 at 10:58

1 Answer 1

2

I think you are best off with a separate service and queue stack for this, rather than making it all in memory. Definitely don't try and run the service within IIS

  • Pick your queuing stack, since you are in .net MSMQ might be easiest as there is nothing else to install. Or you can implement a simple queue on an sql table.

  • create a new 'service' a windows service works well, but again you can make it a console app run on a schedule if that's easier

The service, just deals with a single item. and must be programmed to run at the same time as itself

  • pick next item off the queue and mark it in progress
  • start processing the item loop
  • update percentage complete
  • check for cancellation request
  • finish processing
  • mark queue item complete
  • get next item (or quit console app)

Your website, can then add items to the queue, report on the queue and status of each task by querying the sql database.

Run the service on a separate box to your website, so it wont slow the website down.

If you have spare CPU capacity on the service box, you can add another instance of the service.

If you need more processing power, you can add another box running the service.

This avoids you having to think about async processing of the files in code as the number of concurrent tasks is controlled by running multiple instances of the service app rather than from within the code. It also means that a crashed instance only breaks a single item.

You can upgrade the service to do more than one item asynchronously. But ideally this isn't needed, as you want to process each item as fast as the computer can handle.

Processing two items at a time on the same computer is only useful if you are slowed down by some external factor, say an api call over the internet or your processing of a single file inst helped by more threads/cpus/cores

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.