I need to redesign distributed Publish-Subscribe solution in our product.
Problem description
We have several producers (NT service), queues (MSMQ) and consumers (IIS server). Task handling requires access to the MSSQL DB.
There are two types of tasks in the system:
- ad-hoc tasks, pushed to the queue by ongoing flows.
- batch tasks, every night each producer creates tons of tasks, mostly for maintenance and non real-time processing.
The requirement is to support tens of thousands tasks per day, and to be able to grow. Solution to the problem should be able to orchestrate task execution rate by several conditions:
- Peak-hours - execute small amount of task, doesn't bother the subscribers
- Off-hours - full gas
- DB load - suppose we have metrics that shows how many tasks we can execute
Additional info:
- We thinking to replace MSMQ with distributed queue (maybe KAFKA).
- MSSQL is a bottleneck, subscribers tries to access the same tables, it causes timeouts/deadlocks, but replacement is not an option.
Please advice how to implement the task orchestration.
Should it be on publishers, consumers or separate component?