Change number of publishers/subscribers at runtime

Question

I need to redesign distributed Publish-Subscribe solution in our product.

Problem description

We have several producers (NT service), queues (MSMQ) and consumers (IIS server). Task handling requires access to the MSSQL DB.

There are two types of tasks in the system:

• ad-hoc tasks, pushed to the queue by ongoing flows.
• batch tasks, every night each producer creates tons of tasks, mostly for maintenance and non real-time processing.

The requirement is to support tens of thousands tasks per day, and to be able to grow. Solution to the problem should be able to orchestrate task execution rate by several conditions:

• Peak-hours - execute small amount of task, doesn't bother the subscribers
• Off-hours - full gas
• DB load - suppose we have metrics that shows how many tasks we can execute

Additional info:

• We thinking to replace MSMQ with distributed queue (maybe KAFKA).
• MSSQL is a bottleneck, subscribers tries to access the same tables, it causes timeouts/deadlocks, but replacement is not an option.

Please advice how to implement the task orchestration.

Should it be on publishers, consumers or separate component?

Answer 1

I would have the throttling live on consumer, since your information so far indicates that all consumers should perform similarly, to the same schedule. Any centralized orchestration would have to ask the node what it's usage was, make a basic threshold check, and then turn back around tell the consumer to change itself. No need for the round trip (and commensurate comms code) as this decision is as easily made on the consumer. The consumer can read a shared schedule to know what it's performance profile should be at a given time. A centralized orchestration could be useful if performance profiles needed to change frequently and dynamically for only some nodes. Nothing says that the nodes can't reread the configuration periodically either.

It's an interesting choice to run your consumers on IIS. Do you not have task recovery problems when the worker process recycles unexpectedly? I have not had much success running long-term loads on IIS. However, I assume it is more convenient to deploy that way.

MSMQ is slow, but as you say, it is not your bottleneck. If it were possible, I might suggest that you put a service API in front of your database access so that you could control concurrent access a bit more (maybe even serialize access to avoid locking problems). However, being a legacy app, I assume that queries are executed directly by consumers. Even then I'm not sure it would be enough if you have long-running sprocs. You can, of course, take standard measures like analyzing whether indexes would be helpful to add (or remove).

Answer 2

OK, my reading of your question is that you have messages on MSMQ which say 'do this task', when the worker picks it up that's basically all it says and the worker has to goto the DB to grab information and update stuff.

In my view this is a misinterpretation of the queue worker pattern, you're not really benefiting from having multiple workers and a queue, because at the end of the day the work in done on a single machine.

To make it really scalable, you need to put the full information needed to do the job in the message and understand how the job behaves when other similar jobs are being done at the same time, making it as atomic as possible.

Say you job is to grab some sales data from a transactional DB and compile a report, which means updating another table with some aggregate data. The message should hold the all the input 'sales data' and the output needs to be able to update that aggregate table and have it contain the correct result even it it runs twice, or in the wrong order etc.

In practical terms this may mean you need to break down and re-factor your jobs. You might have :

• Job A, copy the days sales into a temp file
• Job B, read the temp file and output another file of calculated results
• Job C, read the aggregates results and work out if they can be correctly applied to the report tables

DBs are (almost) always the bottleneck, You have to move all the logic out of the DB, no big sprocs or SSIS packages. Just simple crud transactions on known size chunks of data.

A more complicated Queuing system will help with the orchestration of your jobs to a degree, but really it shouldn't be necessary. Your worker machines can run 100% 24/7 because that's all they do and however hard they work they don't affect the performance of the rest of the system