0

I am designing the API architecture for my client:

My client actually exchange huge files (csv) with his Partner with MFT ( SFTP )

  • **Partner ==> MFT ==> Client **

The requirement is to move to API REST exchange mode to replace MFT ( csv file)

  • **Partner ==> API ==> Client consume GET API **

For example :

The partner of my client send every day 200 MB of text data ( csv file 200 000 lines)

Actually my client receives this by MFT ,he asks me if it is best practise to replace this by API REST CALL to transfer large data ( order of 200MB)

I know the best practice for a normal payload is in order of 1 MB ( json payload) for one call HTTP, so I will be doing pagination (200 Pages) and handle errors and so on.

  • Is it a best practice to do this by API?
  • Are there detailed / practical arguments for this ( why we should move to API in this case , my client always asks me this )?
  • Are there techniques to reduce pagination while transfer data 200 MB ( I estimated 200 pages and 200 http call to get all data ) with API while keeping performance ?
4
  • I am so SORRY ! My english is not very good !! I am sorry , I updated my post with more structural words ! – Benhassine Mohamed Jun 3 at 8:18
  • Your english is good enough, your question is quite understable ;-) But using capital letters in social media (apart from certain abbreviations) is often perceived as shouting/ yelling. – Doc Brown Jun 3 at 8:34
  • thanks again for your feedback I will take this in consideration for my next post ! – Benhassine Mohamed Jun 3 at 8:42
  • Just another thing: askers on this site are expected to do some research on their own before asking, present what they found and tell us why it did not meet their needs. For example, googling "large files rest api" returns a lot promising results. If none of them could help you, you should edit your question and elaborate a little bit on those. – Doc Brown Jun 3 at 9:01
1

In your diagram it's somewhat unclear in which direction the data flows. Your arrows are bidirectional, does that mean each participant sends as well as receives files? It is also not clear which party initiates the transfer, which party runs a server, and whether data is pulled or pushed.

HTTP, upon which REST services are built, has generally pretty good performance in pull scenarios because that's the primary usage pattern in the web. If it is possible to make files available to be fetched by a normal HTTP request that might be the best solution regarding performance and simplicity. Pagination most likely wouldn't be necessary, and if you use content compression your 200MB CSV files are probably less than 100MB during transfer, which is somewhat big but not impossible using single HTTP requests.

If the partner can't provide the files on a server you need to go for a push model which is likely more complicated and a little slower unless you can transfer the file in a single POST request with compression. This is being used for image uploads on the web all the time, so it's generally well-supported, too.

Note that a RESTful endpoint does not need to be restricted to JSON data, it is perfectly ok to use a text/csv content type, and the actual implementation of this endpoint might be directly in the web server (if it can handle the authorization stuff correctly), so you don't need to go through some application REST framework although that would still be possible.

3
  • Thanks @Hans-Martin , the correct flow data is :Client <==> MFT <==> Partner – Benhassine Mohamed Jun 3 at 9:11
  • thanks @Hans-Martin , the correct flow data is : PARTNER==> MFT ==> MY CLIENT , the client will expose GET API ( exposed by APIGEE) to expose 200MB of txt data and my client will invoke This HTTP GET , How we can get the file in a single GET request with compression without pagination ? if so why all experts recommand pagination to get huge Data from API REST ? Thanks – Benhassine Mohamed Jun 3 at 9:18
  • I'm not an expert and I can't speak for experts, but I suspect that pagination is recommended for list-like data intended to be displayed to a user who might only want to scroll through the first 1-4 pages of data. It allows you to present first results quicky and avoid transferring too much unnecessary data. In your case, the whole data is always needed, and it is prepared ahead of time, so there's no huge delay before data becomes available. – Hans-Martin Mosner Jun 3 at 9:24
0

Is it a best practice to do this by API

No, one way to see it is, part of REST is having caches between client and server. Transferring file in one way means no caching is needed, so REST is not needed to do this.

You don't want to have a custom API for transferring file. There are many data integration tools that will just work with FTP, SFTP, FTPS, etc. to do this with little effort.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.