Micro-services architecture for Data Ingestion/Transformation pipeline project

Question

I am working on designing a brand new Data Ingestion Pipeline with the Key highlights of the new project are as follows:

• Download and Update data to/from SharePoint using SharePoint APIs
• Download and Update data to/from JIRA/incident management application using JIRA APIs
• Download and Update data to/from SQL sources using provided APIs
• Download and Update data to/from external custom applications using APIs

I am considering the micro-services architecture for the above project where I will be looking at creating 4 separate services for each of the above purpose.

And finally, a batch processing client that would execute all these API services using C# .NET

But I have been wondering if implementing micro-services architecture will be overkill, and rather all I really need is a single client calling all these APIs directly without having to create above individual services.

And additionally, regarding setting up the project in Visual Studio, should all these services be in their own separate solutions or rather be a part of one .NET solution with multiple projects in it.

Answer 1

Micro-services architecture

is the architecture in question.

There are 4 different tasks:

• Download and Update data to/from SharePoint using SharePoint APIs
• Download and Update data to/from JIRA/incident management application using JIRA APIs
• Download and Update data to/from SQL sources using provided APIs
• Download and Update data to/from external custom applications using APIs

And then you write

And finally, a batch processing client that would execute all these API services using C# .NET

The first question which comes to my mind:

• Are the tasks executed in order or out of order?

If they are executed in order: What are the advantages of having "microservice" - whatever this would mean in this context. If the job is to simply gather a bunch of data and stuff it to somewhere else why writing several "services"? And why write "services" at all? This seems to be something which could be "scripted" anyways. Or did you mean Windows services. But calling this "microservice" is - even if they may be small - a bit misleading because typically microservice means some kind of small self contained application with web capabilities baked in. From what you write the "services" you mean provide no service as such. There seems to be no advantage of using your "services" instead of querying the APIs themselves.

The next thing which comes to my mind:

• Why not using "traditional middleware" products like Pentaho (or even BizTalk) or the like?

The scenario you describe is a typical "middleware scenario" from my POV. I would prefer using COTS software over making my hands dirty.

Answer 2

Using micro services only makes sense if you are planning to reuse the services for other applications. Otherwise you would have too much unnecessary overhead. If there is no intention to reuse all or some of the services, you should use a simple component design to encapsulate responsibilities under the hood of one single service API (e.g. REST).

But in general: the nature of micro services is that each service is independent and loosely coupled to other services. This would require them to be individual solutions that can be individually deployed. Each service will likely consist of multiple assemblies (projects)