Should there be a separate service for creating asynchronous job?

Question

Say, our team owns 3 services, one is responsible for creating persons, other is responsible for creating buildings, and 3rd one is responsible for creating jobs. Also, we have one website, which is connected to all these services. Now, we want to add a functionality on our UI that would take a CSV with fields asking values for creating persons, buildings and jobs. Once that CSV is uploaded, it should create persons with some buildings and jobs. Also, say, this kind of functionality is required by some client calling our service from their service. We want this CSV upload functionality, at least in UI, to be asynchronous in nature which means we will return some job id to the user/client and user/client can use that to track the progress of job.

Then, is it a good idea to create a service which does this orchestration kind of work, calling our other services and provide the endpoint to the API of that to the client as well as to the UI? Or UI backend should do the orchestration at their side while client should do it at their side?

I have never seen a service whose only purpose is to create jobs and call other services. What do you suggest and why?

Answer 1

Think of the SOLID Interface segregation principle (not exactly the situation here, but kind of close): you want to limit what the UI can access, especially if you are dealing with something that will be exposed over the internet. You also want the API used by the front end to return specific information pertinent and arranged in a way that is useful to the UI. So, depending on complexity, it is good practice to have a middleware or gateway of sorts between your front-end apps and your back-end apps.

Now, in terms of the orchestrator, yes, the CQRS pattern uses something like this. I don't know if you are applying the pattern, but also for separation of concerns, it'll be better to have this responsibility isolated to a specific application.

So, what I would do is:

1. Have a frontend API dedicated to it. This will serve as a passthrough to your backend apis.
2. Have a separated job for orchestrating the creation of persons with jobs and buildings
3. Have a separated API for the consumption of data as the result of the work of the orchestrator.
4. And if you want to be completely strict in applying Microservices and CQRS (and I am NOT saying you should, it is just a suggestion), the orchestrator will have its own database, different from the other API. Yes, you'll need some sort of additional process to replicate the data.

To answer your other question: Your UI backend/API should NOT do the orchestration, simply because it is not its job, it is outside of its domain.

Answer 2

I work on a system which does have a service that creates tasks which are run by a task manager (on a separate machine).

The system also has a page which orchestrates several API requests for work which could have been run as a task/tasks.

Both approaches have their issues.

Running the task on the back-end opens the door to denial of service caused by a mal actor placing hundreds of task requests. That isn't too hard to imagine. A nice actor submits a task, the task is registered, but the actor fails to recieve a reply, so it submits a new task. There are ways to moderate this, but the problem is there.

Running the task on the front-end opens the door to data corruption. This is because it is entirely possible for the front-end to not finish the task. Even if data corruption doesn't result it can lead to data that is correct but hanging around free floating. Of course there are ways to mitigate against this, but it is still a problem that your system will have to manage.