0

I am looking for a design advice for the following scenario:

Three layer application:

  1. MVC Portal (just frontend app really)
  2. Business logic WebApi (all the magic happens here)
  3. Connected services (just the file processing services here)

Portal is not aware of any database, processing services etc. It only communicates with Business WebApi

So, the business scenario is

  1. User uploads a file into Portal
  2. Portal sends it to WebApi
  3. File is evaluated, a 'project' is created for it in the database etc.
  4. Workflow is built based on the result of the file evaluation and perhaps some project settings as well.

Workflow consists of steps, which are essentially processing tasks handled by the connected services (the 3rd layer).

For example:

  • Step1 - CleanerService
  • Step2 - PreprocessorService
  • Step3 - PostprocessorService
  • Step4 - PublisherService

Each of these steps will take quite a lot of time, so instead of awaiting actively the result I will just post a request and return an immediate result (accepted for processing).

Then, once a service is done with the file, it will send a callback to Business WebApi. Most likely, back to the controller that the request came from.

I will have a simple WorkflowManager class that will lookup the database for workflow for a given file, get a next step ID, figure out the service type and send the file to this service.

Here's where the actual question begins

How do I nicely manage a communication between each of the service controllers and a WorkflowManager?

For example, CleanerService controller (in para code;) )

public class CleanerServiceController : ApiController
{
    [HttpPost]
    public async Task<StepTriggerResult> SendCleaningJob(File file)
    {
        // this action is called by the Portal
        // and it uses its HttpClient to send another Post to a proper service
    }

    [HttpPost]
    public async Task CleaningCompleteCallback(File file)
    {
         //I could maybe call a workflow manager here e.g.
         this.workflowManager.StartNextStep(file);
         // the start next step would figure out the controller for the next step service and call action similar to SendCleaningJob(file);
    }
}

However, adding the workflowManager to the controllers for the actions seems a bit hacky. I think it would be better to have it centralized somehow, rather than scattered all across the controllers. Perhaps a shared api controller for all callbacks would be better.

Any ideas?

1

"File is evaluated, a 'project' is created for it in the database etc."

By your own admission, the business logic is tightly coupled to your database. I know this is not related to your actual question, but it is something you should be aware of. In my opinion, it is better for the front end to be aware of the datalayer than having the business logic aware of it. It is very difficult to decouple business logic and data access once they are tightly coupled.

This brings me to a question I have for you, do you have a compelling reason to place your business logic in an web API instead of a dll/sll? You may very well, but your question has not made it clear that you do.

Placing your business logic in a web api creates significant engineering overhead, including, but not limited to the problem you are asking about here.

If your business layer were not web based, you would be able to easily coordinate steps using events.

Also, layer two can be called asynchronously, there is no compelling reason to call layer 3 asynchronously. You can simply wait for the steps to complete with synchronous calls from 2 to 3, as long as layer 1 is allowed to continue.

You could also use the claim check pattern; you assign each request an ID, then continue. Once the cleaner service completes, it calls back to layer 2 with the ID. But again, I do not see how this is necessary.

  • The division into three layers is not my decision actually. It was partially the client's request (they want everything except the portal to be behind a firewall), and partially because the same webapi /backend solution could then be used for a different client portal etc. – Bartosz May 31 '17 at 19:01
  • Also - indeed, the business logic is coupled to the database, its the only layer that communicates with the database. Can you please elaborate on how is that a problem, or point me to some reading on this? – Bartosz May 31 '17 at 19:03
  • You are confusing tiers with layers. A layer only presumes logical isolation, a tier is system level isolation. You are building a three tier system with no perceivable benefit from the overhead (unless you count going way overbudget as a benefit). You should start simple, 1 teir, 3 layers. 1. Frontend layer 2. data 3. business. Data references business. frontend references data and business, business has no knowledge of either – TheCatWhisperer May 31 '17 at 19:13
  • If you have performance problems, or the persistence method needs to change, having tightly coupled business-dataaccess will make it so you might as well rewrite the entire application. – TheCatWhisperer May 31 '17 at 19:15
1

Took me a minute to digest your architecture. It's unusual, and I see a few problems with it.

  1. You have an application stack that strays from the typical master-slave relationship. In some cases, the WebApi calls the service, and in others the service calls the API. Normally, calls are always initiated by the upper level in the stack and the lower level can only call levels below it.

    Why is this an issue? Well consider the case where you need to add a new bit of logic and a new callback. In order to deploy the new service, you have to have the new WebAPI in place in order to support the callback, and in order to deploy the WebAPI, you have to have the new service in place. This sort of circular relationship can make upgrade and deployment scenarios very painful, because everything has to be done in lockstep.

  2. You have long running transactions running in a service and a workflow that depends on a real-time callback. Think about your failure modes and you may start wondering how you will handle them. What if a callback fails due to a transient network problem? What if you have to recycle your services while a long running transaction is running? What if your processors are too busy to handle any new files at the moment? Web stuff isn't good for this.

I would suggest a slightly different approach

  1. Web application ("portal") simply for uploading files. This could even be a simple two-tier application, no service layer would really be called for.

  2. One or more daemons, scheduled tasks, or Windows services that look for uploaded files, move them to a location where they can be worked on (into the database if you really want, although I'd prefer flat files on a SAN personally), and creates and initializes the workflows for them.

  3. A database structure that defines a workflow in flight and which steps have already been completed. Each workflow should have a unique identifier.

  4. A queue structure (e.g. a database table) that indicates which workflows reserved for processing, in the process of being processed, or recently released from processing.

  5. Another task/service that reads the queue, reserves files for processing, and processes them. You need to reserve them first because you may end up with more than one processor running, e.g. if you need to scale out.

  6. A recovery mechanism that allows stale queue items to be requeued. This will be handy if the system ever goes down and you need to recover.

  7. Monitoring tools to check on the status of the services, the size of the queue, and NoC alerts if services fail or the queue gets too long.

This would be a much more robust approach. Windows services are not as sexy as web APIs but they are a better tool for the problem.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.