Is a message queue the right pattern to use for intra-module communication in a .net core application

Question

Background Information

We have a .Net core application that uses Hangfire (pretty version of cron) jobs to batch process different types of data.

As a bogus example, we have one job that will run every 30 minutes to query an external source to get a list of products and saves them into a local database. (for discussion purposes, let's say the table is called "ProductList"

Then we have another job that runs every X minutes that will add more information about products in the same local database based on some business logic. (ProductVendorLocations)

The job that checks for the latest product list could actually end up creating or deleting existing products from the master list. if a product is deleted, we need to remove it from the ProductVendorLocations table let's say. For some reason unbeknownst to me, we have this logic in a separate job. It smells like it should be something done sequentially - meaning as soon as we delete a product, the code should immediately remove it from the related table as well. I've currently raised this as an issue but until they change that I have a question about how to handle triggering "dependant" jobs. We have many jobs that we run ... many of them tied to the master product list.

Questions

In a micro-services architecture, one way that disparate services communicate change to each other is using a message bus. So I'm wondering if that could be a pattern to employ inside my monolithic app? Install Rabbit MQ or something and have different jobs notify each other just to say "i deleted a product. anyone who cares, do what you will" type thing.

Alternatively, I was thinking about creating a policies (or sagas??) folder in my solution. I would create a class / policy called "ProductDeletionPolicy".
Then I would do something like this in my hangfire job that synchronizes the master product list:

   LatestProducts = getLatestProductListfromUpstream()
   for each product in LatestProducts
         existsAlready = doesItExistLocally()
         if existsAlready then
               if product.hasChanges then saveChanges(product)
               if product.isDeleted then
                  ProductDeletionPolicy.Delete(product)
               end if
         else
               addNewProduct()
         end if
            
   loop

The ProductDeletionPolicy would have to list all the methods that would need to be run when a product is deleted. Something like this:

   class ProductDeletionPolicy
   {
        public bool Delete(Product product)
         {
              //list the methods in different modules that need to be run?
             productInventories.MarkAsDiscounted()
             productVendors.NotifyAboutDiscontinuedProduct()
            etc, 
         }

   }

I hesitate to go the mq route because we are in the throws of retraining the team to adopt Clean Architecture and Domain Driven Design. So i'd rather not introduce something net new again. But I'd like to know what the best option is as far as a pattern to follow.

Thanks for reading.

Answer 1

It's a tempting pattern to follow but it's easy to over do it and land yourself in a world of orchestration pain.

There are good reasons to have "out of process" jobs run on a different machine. Your product update job is a good example.

You want to keep upto date with the external data, but it takes a long time to run and you only want to run one of these jobs at a time. It makes sense to separate it out and have its own program that does the work on its own CPU resources so it cant interfere with the main application.

However, once you have that worker process app. It doesn't necessarily make sense to have another, separate, one that does the "get extra info for new products" process. You can just add this logic to the existing process. It only runs every 30min, its on its own box at the back of the server room, its fine.

If you do separate everything out, add the queues to allow them to all message each other when events happen etc it might seem neater, but its the exact same code, you've just replaced function calls with message queues and networking. Which means you still have the same debugging issues, just now you have to work out what's listening to what queue and it can soon becomes very complicated.

MQ events that push to other processes are a great thing. But you should always be looking to minimise them where possible to reduce your orchestration burden.