0

I am in the process of carrying out a software architecture for my client, in fact

THE HISTORY OF THE REQUIREMENT :

MY PARTNER SEND cvs files via MFT to MY CLIENT , many times a day .

MY CLIENT has a listener ( cron job in linux and batch ) , when a file is received in a folder , the client consumes it and store it in database ( SQL SERVER)

Regarding data , the estimated trafic : 700 files a day Millions of lines in cvs file to be consumed there is no order in data , each time a file is received it is processed by batch .

The new solution is to transfer data via API REST CALL and not OLD MFT :

The need is to retrieve huge data from a partner by calling API REST (JSON) to finally store the result in our SQL database (not too much processing to do in between)

I proposed two solutions:

SOLUTION 1 : Event Driven Architecture and PARTNER EXPOSE API REST TO RETRIEVE DATA :

THERE ARE STEPS OF THIS SOLUTION :

OUR PARTNER : create an event in our BROKER QUEUE ( Message indicates that there is new data to get : name API RESSOURCE TO CALL / parameters / Bussines Domain )

MY CLIENT : GET the message and read it from the QUEUE

MY CLIENT : Based on this message , my client CALL the REST API ( GET exposed by our PARTNER ) to GET THE DATA FROM JSON RESPONSE of the call HTTP ( this will be implemented for example by spring batch )

MY CLIENT : STORE The result of this huge response on SQL DATABASE without many modification

SOLUTION 2 : THE PARTNER DO NOT EXPOSE API , MY CLIENT WILL DO THIS !

THERE ARE STEPS OF THISE SOLUTION :

MY CLIENT ==> EXPOSE A REST API to our PARTNER : THIS ENDPOINT IS a POST HTTP OPERATION

OUR PARTNER ==> THIS time , our Partner will call THIS POST API REST AND send all data in REQUEST BODY , as a result , the partner update our SQL DATABASE when he calls our API REST ( CLIENT CALL THE POST API WHEN HE WANTS TO UPDATE DATA)

MY Notes :

THE PARTNER IS OPEN to ALL OUR SUGGESTIONS ( SOLUTION 1 or 2 ) and we estimates that it there will be huge data transfers between Partner / Client

My requirements : evaluates the pros and cons of these solutions

For my part, I opt for the SOLUTION 1 because :

pos1 : the processing will be asynchronous pos2 : we will be the master of our data to be added to our database ( is it good as an argument ? ) pos3 : processing in almost real time cons1 : architecture is complex when in we move to cloud cons2 : Partner is not familiar perhaps with this solution

The second solution ( we expose API , the partner simply call it ) pos1: is rather light (less expensive) it is an advantage, cons1 : but has the disadvantage that the processing will be done synchronously cons2 : that we will not be master of the data to add to our REF (unless checks to be made in the POST action to be exposed) cons3 : call of API will be an additional charge by our partner

Thank you for your feedbacks : (advantages / disadvantages) Regards

0

Personally, I'd lean towards solution 2. The reason for that is that it's easier to see that the responsibility to send the data to CLIENT rests with PARTNER. It's also easier for PARTNER to confirm delivery of data.

With solution 1, PARTNER needs to keep track of which batches have been messaged to CLIENT but not yet picked up by CLIENT. Then PARTNER must have some routine which checks for "stale" batches, which haven't been picked up after some amount of time. Possibly do a re-message, or other remediation.

With solution 2, include some checksum or total record in the data, then CLIENT can recalculate and confirm correct submission of data. Then respond with appropriate HTTP Response code. So PARTNER has instant confirmation that batch received and no need to track stale messages.

There's no reason that the REST API has to import the data into the SQL database directly. It would probably be sufficient to drop it into the same linux folder that the current process is using. Then the cron job could continue to work as it always has.

IMHO, the idea that you are losing control over the data import is a red herring. The function here is the file transfer, and a POST from the PARTNER does exactly that in the most direct manner possible.

1
  • Thanks Dave for this détails , It is very helpfull to pick the right choice – Benhassine Mohamed May 25 at 14:25
2

First of all I would definitely agree with @Hans-Martin Mosner - the best integration starts with a good communication. Adapt the solution so it is good for both parties - your team and the partner.

The second general advice is that one size doesn't fit all. You need to consider your specific context - the importance of the integration, the frequency of the integration (if it is repeatable?), the volume of the integration. There is no silver bullet.

The third general advice is to ask your team what they know the most, feel comfortable with (or want to learn in some cases). It is super easy to adopt any solution... in the wrong way.

Now, considering that we don't have full information about your problem, a few aspects to think about.

  1. Do I need an order guarantee?

As it is quite easy to force a queue/broker to act as a FIFO (but ensure you understand the details). It is definitely not easy to do it when you have direct communication of several (assuming some scaling) independent rest consumers trying to save the new message.

  1. Do I need a persistence guarantee?

In this case the REST version is pretty easy - you either get OK response status 2XX or some error 4xx, 5xx in which case the partner can retry. The same scenario can be handled with the queue/broker but in most cases requires extra logic and can lead to tricky corner cases.

  1. Performance

Regarding the performance of the solutions both can be pretty similar. What can matter here is the actual load. In some extreme scale (e.g. some IoT measurements) scenarios brokers (e.g. Kafka) may be able to read amount of data which would be hard to handle with reasonable number of web endpoints instances (there is no magic here, just the way we divide and conquer the problem). At the same time it doesn't mean you will have data saved faster (it may be even the opposite). You can also use an asynchronous approach with the REST version (by returning the status just after receiving a message) but it is of course a compromise on the persistence guarantees.

Note:

The size of the message matters as well. The HTTP can have high latency with very big messages (can be handled with partial delivery). At the same time different MQ/Brokers have different limits on the message size. In some extreme scenarios you may think about other alternatives to these two.

  1. How do I expose the solution?

It is fairly easy to expose the http endpoint. It is not so easy to do the same with MQ/Broker.

  1. Can I handle a new component in my infrastructure?

In most cases the less the better. Of course unless you need it. Think twice.

  1. Is the message contract stable and constant?

A versioning of http endpoints is pretty standard and you can find plenty of examples. The versioning with MQ/Broker + external may be tricky.

Above are just example hints (there are definately more). There are pros and cons for both. Both can be equally valid in most reasonable boundaries scenarios. Both solutions are used with success - first mostly as incoming webhooks and latter as a queue communication, e.g. in any streaming processing. Hope that they can help you with your selection.

1
  • Thanks Tomasz for your detailed response ! , the problem is that both client and partner are not sure what the require , their only goal is to move from MFT ( file transfer ) csv file to API REST CALLS because it is the the trend , I updated my post because I think my presentation of the requirement was not very clear , could you please review this last version ? – Benhassine Mohamed May 24 at 11:13
1

In my (limited) experience, it was helpful to check which of the two sides is less flexible, and to accommodate their needs at least to some extent. If you're able to set up a variety of solutions on your side (REST API to accept the data, API to accept an event and retrieve the data at your convenience, RabbitMQ or other message queue on either side, ...,) then asking the partner about their preferences might lead to a workable solution quicker than if you choose an interface that is "best" according to some metric but with which the partner is unfamiliar.

At the moment, I would consider a message queue but that may well be due to the fact that RabbitMQ is currently being evaluated in a project, so my head is full of that right now.

1
  • Thank you Hans-Martin , a good solution would be a marriage between partner capacity to hold the solution ( familiar with tech teck ) and our expertise , but here the partner is open to all our suggestions ( he does not have any expertise ) – Benhassine Mohamed May 24 at 10:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.