3

I'm working on a portion of an ERP system where I need to process data in a way that's similar to a series of batch jobs, and I'm struggling with deciding the best program architecture to use. I'm asking here because I know I'm ignorant of a lot of state-of-the-art programming methods, and I'm worried about designing my software in an archaic way without realizing it.

My system needs to process batches of data and have the processing jobs be able to be scheduled at regular intervals, and also to be run on demand. Some of the processing also relies on outside web services.

Right now my architecture is that I have one single Visual Studio C# solution (with several projects inside of it). The solution produces a program which has an interface for running jobs on demand also for configuring a schedule for the jobs to run automatically. The single program also has all of the batch processing code in it.

Within the program, each "batch job" is called by calling a method on my main controller class, such as public async Task BillOrders(), which then calls the appropriate method from a high-level service class, which calls lower-level services, and so on. The classes themselves are all pretty well separated in terms of the single responsibility principle. The main controller calls each batch job method asynchronously and handles monitoring them, reporting errors, throttling, and so on.

This all works, but I'm worried that this isn't the way I should be doing it. Specifically, my concerns are:

  1. Since I have several separate batch-process-type functions all in the same program, I'm worried that a crash or bug in the program could cause all the batch processes to stop working. It also seems bad to have one single program that does a bunch of different things.

  2. Even though each batch process is related to one area of my ERP, I'm worried that having a single codebase with different functions in it will end up creating something monolithic and difficult for several developers to work on different aspects of simultaneously.

One of the main reasons why I've done this as a single program (and Visual Studio solution) is my Entity Framework context and service classes are shared between each batch job. If I split the program into several smaller programs, each program would have a lot of classes and the EF context that are the same code. And any bugfix or extension to a service in one program would need to be copied to the others. Also, my EF context has several dozen tables that are done with Fluent API.

I considered making a series of microservices, but there are so many cross cutting concerns that each microservice would need a lot of the same EF context and service classes that I described above, thereby defeating the goal of using microservices.

Is there a commonly accepted architecture for this sort of thing? Is what I'm doing OK, or am I programming as if it's 15 years ago?

  • Are you designing an ERP made of pieces with batch jobs as a glue ? If it's the case, and sorry to be direct, you'd be designing as if it's 25 years ago. First question: why not design the flows between the pieces real time ? And if you're considering microservice (which would really be state of the art), why not organize your information flows on an event-stream, to which all your services could subscribe ? – Christophe Sep 21 '18 at 21:38
  • I'm not sure what you mean by "design the flows between the pieces real time". The main reason that I'm hesitant to use microservices is that each batch job, which is incremented as a single service, would use a lot of the same service/EF code. If I split out the common services (for instance services to retrieve various bits of information about orders), I would wind up with so many microservices that communication and speed between them would become an issue. Is there a good way to share the code for an Entity Framework DbContext between all microservices? – Ben Rubin Sep 21 '18 at 21:44
  • 2
    Let's take an example with SAP: there's a purchase module handling purchase orders, a logistic order handling inventory, and an accounting module handling invoices. There is no batch job that tells the inventory module how much new inventories are expected to be received. And when the pieces arrive, there's no batch job to tell the accounting module how much the inventory is worth. It's all done "real time", i.e. every single transaction in any module will automatically and immediately update all the other modules. The alternative is batch jobs and delays to get up-to-date infos. – Christophe Sep 21 '18 at 21:54
  • 1
    IMO, this is not necessarily suited for microservices arch. style, which is more about supporting extreme levels of scalability required by services like Amazon and Facebook, and arguably requires infrastructure supporting CI and CD, and things like Event Sourcing / event streams, design based on bounded contexts, separated databases, eventual consistency, etc. (See this.) – Filip Milovanović Sep 21 '18 at 23:50
  • 1
    My first instinct, based on your description, would be to initially go for some form of pipes-and-filters architecture (which yes, is ages old, but that's not an issue), and move to full-blown microservices style later if necessary. This will introduce some extra complexity, so make sure to weigh the pros and cons. You'll also have to think about what can be processed independently and in parallel, and what cannot, what must be atomic, how to communicate between filters/processors, etc. – Filip Milovanović Sep 21 '18 at 23:51
2

Since I have several separate batch-process-type functions all in the same program, I'm worried that a crash or bug in the program could cause all the batch processes to stop working.

Are you running multiple jobs at the same time in the same process? In that case there is a bit of a risk that one error might terminate other jobs. This will only happen for errors which damage the whole process. This is fairly rare. Examples are out of memory conditions, stack overflows or people terminating the process. Otherwise, you can wrap your processing code in try-catch to isolate them from one another. This is what web applications do and normally it works very well.

You could also run each job as a separate OS process. Then, the jobs are totally isolated. You can reuse the same EXE file for this. Make an integer or string command line argument that tells the EXE which of the many possible jobs it is supposed to launch.

Reusing the same EXE file saves you from the need to create many C# projects and have the code management overhead that would come with those projects.

It also seems bad to have one single program that does a bunch of different things.

What is your concern? Maybe you are worried that this might go against the single responsibility principle. This principle is about the same code doing multiple things. As I understand it your jobs are fairly isolated from one another. Also consider that typical web application are doing 100s or 1000s of different things and it's not an issue. The SRP just a principle, not a hard rule. Do what works for you.

Even though each batch process is related to one area of my ERP, I'm worried that having a single codebase with different functions in it will end up creating something monolithic and difficult for several developers to work on different aspects of simultaneously.

If you are concerned the codebase might become unwieldy then I'd argue that splitting the code into multiple projects would not gain you anything. It's still the same code, the same dependencies and architecture in a different place.

Solving entanglement is never possible just by putting code into a different place (additional projects or services). You must make architectural changes.

So whether you build one EXE/microservice per job or if you make each job merely it's own C# class is not much of an architectural difference. It just adds a lot of development complexity.

Since your jobs seam fairly isolated from one another I'm not concerned about putting all of them in the same project.

It also sounds like you want to share some infrastructure such as Entity Framework models. That's also a good reason to keep things together.

  • 1
    Thank you for the answer. You did a great job of addressing my concerns. Regarding your first point about running multiple instances of the same .exe, that's an interesting solution. I hadn't though of that. – Ben Rubin Sep 22 '18 at 13:26
1

Your domain is so different from mine and the comments below your question from experts in this domain are going way over my head, but with "batch processing" and your two specific questions related to your very immediate design concerns, immediately I think of the command pattern and IPC (interprocess communication) as my goto techniques for solving these issues.

Even though each batch process is related to one area of my ERP, I'm worried that having a single codebase with different functions in it will end up creating something monolithic and difficult for several developers to work on different aspects of simultaneously.

A solution for that is to replace your central "main controller" interface with decentralized concrete command objects which all implement the same abstract Command interface. You can execute them asynchronously from a central place if that's a need. You can also combine that with a factory to instantiate commands by a string indicating the name which could be useful for the second point below. This way you can keep adding new commands to your system non-intrusively without some central interface growing to become a monolith.

Since I have several separate batch-process-type functions all in the same program, I'm worried that a crash or bug in the program could cause all the batch processes to stop working. It also seems bad to have one single program that does a bunch of different things.

The solution for that if you want something really robust is spawn a separate process to execute these commands (or spawn it in advance and communicate with an existing one to tell it new things to do with IPC). You don't have to program separate binaries for every single possible task/job (command) that could be performed as usr pointed out. You can have one generalized command executor binary or even reuse the same one you already have now with command arguments supplied to tell it what command to execute, and some form of IPC (there are many options here from named pipes to file mappings to mailslots and others on Windows -- I've generalized the file mapping and shared memory method as the one most generalized for my purposes, though it requires semaphores) to share relevant state between two or more processes. If you do it that way, then if some sub-process crashes, it won't bring the host application performing these batch jobs down with it in flames.

I believe that's how many of our browsers work to avoid crashing when Flash or some JavaScript or WebGL crashes or something like that. They likely spawn these things in separate processes, and you can see with Chrome at least that it has its own task manager for that (shift+esc) and that it tends to launch many instances of Chrome.exe independently of how many Chrome instances you opened yourself.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.