0

Background Information

We have a .Net core application that uses Hangfire (pretty version of cron) jobs to batch process different types of data.

As a bogus example, we have one job that will run every 30 minutes to query an external source to get a list of products and saves them into a local database. (for discussion purposes, let's say the table is called "ProductList"

Then we have another job that runs every X minutes that will add more information about products in the same local database based on some business logic. (ProductVendorLocations)

The job that checks for the latest product list could actually end up creating or deleting existing products from the master list. if a product is deleted, we need to remove it from the ProductVendorLocations table let's say. For some reason unbeknownst to me, we have this logic in a separate job. It smells like it should be something done sequentially - meaning as soon as we delete a product, the code should immediately remove it from the related table as well. I've currently raised this as an issue but until they change that I have a question about how to handle triggering "dependant" jobs. We have many jobs that we run ... many of them tied to the master product list.

Questions

In a micro-services architecture, one way that disparate services communicate change to each other is using a message bus. So I'm wondering if that could be a pattern to employ inside my monolithic app? Install Rabbit MQ or something and have different jobs notify each other just to say "i deleted a product. anyone who cares, do what you will" type thing.

Alternatively, I was thinking about creating a policies (or sagas??) folder in my solution. I would create a class / policy called "ProductDeletionPolicy".
Then I would do something like this in my hangfire job that synchronizes the master product list:

   LatestProducts = getLatestProductListfromUpstream()
   for each product in LatestProducts
         existsAlready = doesItExistLocally()
         if existsAlready then
               if product.hasChanges then saveChanges(product)
               if product.isDeleted then
                  ProductDeletionPolicy.Delete(product)
               end if
         else
               addNewProduct()
         end if
            
   loop

The ProductDeletionPolicy would have to list all the methods that would need to be run when a product is deleted. Something like this:

   class ProductDeletionPolicy
   {
        public bool Delete(Product product)
         {
              //list the methods in different modules that need to be run?
             productInventories.MarkAsDiscounted()
             productVendors.NotifyAboutDiscontinuedProduct()
            etc, 
         }

   }

I hesitate to go the mq route because we are in the throws of retraining the team to adopt Clean Architecture and Domain Driven Design. So i'd rather not introduce something net new again. But I'd like to know what the best option is as far as a pattern to follow.

Thanks for reading.

13
  • 1
    You wrote "monolithic app", which makes me wonder if your jobs are running in different processes? But to give you a quick answer: an MQ is one standard way for interprocess communication, regardless which other technology your are using (you use it for services implemted in several different technologies). But for a single process application, an MQ might be overengineered.
    – Doc Brown
    Commented Jun 6, 2023 at 19:55
  • @DocBrown we spin up different threads for each "job" that is triggered. monolithic in the sense that we have one .net core project that has many different cron jobs that can be triggered.
    – dot
    Commented Jun 6, 2023 at 20:01
  • 2
    I'm wondering if this isn't just a mediator pattern if you're connecting things that exist in the same runtime.
    – Flater
    Commented Jun 7, 2023 at 3:39
  • I inherited a product that did this with its own MQ, and it's fairly nasty; if something is inside the same process, and it's at all feasible, implement your dependent triggers as delegates and run them at the same time.
    – pjc50
    Commented Jun 7, 2023 at 9:32
  • 1
    you dont want to make mq work like function calls, you want to take the function calls out and make the background jobs work like background jobs
    – Ewan
    Commented Jun 8, 2023 at 13:08

1 Answer 1

2

It's a tempting pattern to follow but it's easy to over do it and land yourself in a world of orchestration pain.

There are good reasons to have "out of process" jobs run on a different machine. Your product update job is a good example.

You want to keep upto date with the external data, but it takes a long time to run and you only want to run one of these jobs at a time. It makes sense to separate it out and have its own program that does the work on its own CPU resources so it cant interfere with the main application.

However, once you have that worker process app. It doesn't necessarily make sense to have another, separate, one that does the "get extra info for new products" process. You can just add this logic to the existing process. It only runs every 30min, its on its own box at the back of the server room, its fine.

If you do separate everything out, add the queues to allow them to all message each other when events happen etc it might seem neater, but its the exact same code, you've just replaced function calls with message queues and networking. Which means you still have the same debugging issues, just now you have to work out what's listening to what queue and it can soon becomes very complicated.

MQ events that push to other processes are a great thing. But you should always be looking to minimise them where possible to reduce your orchestration burden.

3
  • So a few comments. First. Thank you! Second. for now everything has to stay on a single massive server. And nothing can be containerized just yet either. i think we would have about 4 or 5 key Hangfire jobs that would need to listen and react to specific events or messages. Does this change any part of your response?
    – dot
    Commented Jun 7, 2023 at 18:25
  • 1
    not really, it makes it even more questionable as to why you want to use a MQ when everything is one process anyway
    – Ewan
    Commented Jun 7, 2023 at 19:14
  • like you create a background worker to run in the background. If you add a way of calling it you might as well have just called a function and not had a background worker!
    – Ewan
    Commented Jun 7, 2023 at 19:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.