2

Background:

We are in the process of converting a traditional multi-service Windows Service (WS) application into an Azure Service Fabric microservices grid running on a Service Bus messaging layer.

In the WS application, the "data loader" service was a singleton to prevent duplication of effort with regards to loading data into a DB from external sources as part of a job.

In the new grid (and because we're using SQL Server 2016 with it's parallelised bulk data import functionality), we would like to have multiple instances of the "data loader" service but it is still a standard scenario that two or more jobs with dependencies on the same data set may be pushed onto the processing grid at one time.

Question:

Is there a standard pattern for ensuring that the duplication of effort with regards to loading specific sets of data in this type of architecture is avoided?

  • 1
    Would it be a problem to have a centralized "data import job assigner" that assigns the jobs to avoid duplication of effort? Or is the problem that you need to get the loaded data into multiple databases? – Frank Hileman Mar 11 '17 at 19:44
  • It is a single database that is being loaded into. I think the issue here is the separation of concerns i.t.o SOLID SRP. When I described the service as "checks whether the data has been loaded AND loads it if necessary" it raised a flag in my mind that I should probably separate those functions here. If you add a description of your comment as an answer I'll accept it. – toadflakz Mar 14 '17 at 8:37
2

Create a centralized data import job assigner object that assigns data import jobs to other objects. The jobs may be executed by threads within the same process, or by external processes, if running on multiple computers. The centralized assigner object avoids duplication of effort.

With regards to the Single Responsibility Principle mentioned in your comment: this "principle" is difficult to follow because the word "responsibility" is vague. The definition of a "responsibility" should vary according to the level of abstraction and the specific software to be designed. At the highest level of abstraction, a "responsibility" may encompass a wide range of tasks, while at the lowest level of abstraction, a "responsibility" may be limited and specific.

At the lowest level, some people consider two lines of code to be two separate responsibilities. Having worked with code that reduces all methods to nearly one line, I believe this type of design just as horrible, or worse, than a giant monolithic class or method.

  • 1
    I 100% agree with you that a pragmatic approach is best. – toadflakz Mar 17 '17 at 8:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.