Multi-instance Microservice Grid: Preventing Duplication Of Effort When Resolving Cached Resources

Question

Background:

We are in the process of converting a traditional multi-service Windows Service (WS) application into an Azure Service Fabric microservices grid running on a Service Bus messaging layer.

In the WS application, the "data loader" service was a singleton to prevent duplication of effort with regards to loading data into a DB from external sources as part of a job.

In the new grid (and because we're using SQL Server 2016 with it's parallelised bulk data import functionality), we would like to have multiple instances of the "data loader" service but it is still a standard scenario that two or more jobs with dependencies on the same data set may be pushed onto the processing grid at one time.

Question:

Is there a standard pattern for ensuring that the duplication of effort with regards to loading specific sets of data in this type of architecture is avoided?

Answer 1

Create a centralized data import job assigner object that assigns data import jobs to other objects. The jobs may be executed by threads within the same process, or by external processes, if running on multiple computers. The centralized assigner object avoids duplication of effort.

With regards to the Single Responsibility Principle mentioned in your comment: this "principle" is difficult to follow because the word "responsibility" is vague. The definition of a "responsibility" should vary according to the level of abstraction and the specific software to be designed. At the highest level of abstraction, a "responsibility" may encompass a wide range of tasks, while at the lowest level of abstraction, a "responsibility" may be limited and specific.

At the lowest level, some people consider two lines of code to be two separate responsibilities. Having worked with code that reduces all methods to nearly one line, I believe this type of design just as horrible, or worse, than a giant monolithic class or method.