1

I want to build a middleware system that my company can use and expand on to handle the integration points between our various bespoke and 3rd party systems as well as our hosted websites.

I am having some trouble with the design/architecture and would like some advice (please let me know if there is a better site to ask these questions).

The middleware will basically be the broker between our ERP and our other systems such as our website/3rd party retailers etc. It will be mainly dealing with API calls and XML file exports.

I was thinking of:

  • Using RabbitMQ as a queuing mechanism.
  • Using Quartz.NET as a scheduler contained within a Windows Service. It would be very lightweight and would simply load up a list of jobs from a database, dropping these into the queue when their schedule is reached.
  • Using a windows service (I'd call this the agent) which would be a consumer of the RabbitMQ queue. It would monitor the queue and process the jobs as they would come in. Ideally, I would want to be able to run separate jobs at the same time so that one job doesn't necessarily tie up another job from running. We can always scale this portion out if required.

Does anyone see any issue with this approach? I can't find a lot of example implementations on the internet about using RabbitMQ as a worker service in .NET (using multiple threads) and am not sure if using a scheduler service to simply drop items on the queue is the right approach.

  • Why not ? It allows you to decouple the producer and the consumer. On the other hand, you don't take advantage of the queueing. If it designed to stay like that, one could ask why you do batch processing by batch-loading your queue and not directly process the records from the database without additional overhead. – Christophe Jun 29 '17 at 7:07
  • That's a good point. I suppose my initial thoughts of using a queue were that I could scale out if necessary.. But if I'm keeping track of the jobs in the database anyway.. Its kind of a queuing mechanism in itself. I did think of using a queue to easily dispatch jibs to different queues in the future. For example, all website integrations can happen on this server etc etc. – Lock Jun 29 '17 at 7:10
  • @Lock be mindful of your throughput and performance requirements if you decide to move your queueing responsibilities to the database. The main advantage of an explicit queue is eventual consistency. Many producers can drop messages into the queue asynchronously which can be handled by any number of consuming agents on their own schedule. You run the risk of making your entire application I/O bound if you rely on reads and writes from/to a database at your ingestion point. That is why queue technologies exist. – K. Alan Bates Jun 29 '17 at 14:16
  • Windows has built-in message queuing as well, and it is very reliable. MSMQ. – Frank Hileman Jun 29 '17 at 17:57
1

To elaborate a little bit more on our comment exchanges.

Advantages of your queue based approach

Your queue based approach has definitively some advantages:

  • Producers ("your db feeding jobs") and consumers (your agents) are decoupled: They don't have to know each others internals. They only have to know your queue and the data structure therein. THis will make your whole architecture more maintainable.
  • You could scale the producer side and enrich the data sources by connecting other applications to the queue.
  • You could cope with production peaks, when producer proce fasters than consumers can consume, by spreading the processing over time thantks to the asynchronous behavior allowed by your queue.
  • You could scale the consumer side by multiplying the processing nodes and distribute them across servers, if there'd be any lack of capacity.
  • depending on the queuing system, you could even scale the queue through distribution (With RabbitMQ you can set up a distributed installation, same for other systems like for example Kafka which aims specifically at big data).

Here a picture to highlight all this: enter image description here :

Disadvantage of your queue

The queue adds a level of complexity to your architecture and has some impacts:

  • The queuing system is an additional system to be installed, maintained and operated.
  • Everything is dependent of the queue. New releases could require to update all the producers and consumers. And you might find yourself dependent of your queuing technology vendor.
  • In terms of processing power, the queuing architecture adds some overhead, adding more processes and requiring processing power for the intermediary.

Conclusion

In my opinion, the inconvenience are marginal compared to the benefits. The flexibility and scalability enabled by the queuing architecture makes it a future proof option in many scenarios.

However you don't seem to really take advantage of the queueing in your system. If your architecture is designed to stay like that, one could ask why you do batch processing by batch-loading your queue and not directly process the records from the database without additional overhead. If everything is driven by your db, your db will anyway be the bottleneck.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.