Move from mft to api REST style to get or post large data 200mb , is it best practice?

Question

I am designing the API architecture for my client:

My client actually exchange huge files (csv) with his Partner with MFT ( SFTP )

• **Partner ==> MFT ==> Client **

The requirement is to move to API REST exchange mode to replace MFT ( csv file)

• **Partner ==> API ==> Client consume GET API **

For example :

The partner of my client send every day 200 MB of text data ( csv file 200 000 lines)

Actually my client receives this by MFT ,he asks me if it is best practise to replace this by API REST CALL to transfer large data ( order of 200MB)

I know the best practice for a normal payload is in order of 1 MB ( json payload) for one call HTTP, so I will be doing pagination (200 Pages) and handle errors and so on.

• Is it a best practice to do this by API?
• Are there detailed / practical arguments for this ( why we should move to API in this case , my client always asks me this )?
• Are there techniques to reduce pagination while transfer data 200 MB ( I estimated 200 pages and 200 http call to get all data ) with API while keeping performance ?

Answer 1

In your diagram it's somewhat unclear in which direction the data flows. Your arrows are bidirectional, does that mean each participant sends as well as receives files? It is also not clear which party initiates the transfer, which party runs a server, and whether data is pulled or pushed.

HTTP, upon which REST services are built, has generally pretty good performance in pull scenarios because that's the primary usage pattern in the web. If it is possible to make files available to be fetched by a normal HTTP request that might be the best solution regarding performance and simplicity. Pagination most likely wouldn't be necessary, and if you use content compression your 200MB CSV files are probably less than 100MB during transfer, which is somewhat big but not impossible using single HTTP requests.

If the partner can't provide the files on a server you need to go for a push model which is likely more complicated and a little slower unless you can transfer the file in a single POST request with compression. This is being used for image uploads on the web all the time, so it's generally well-supported, too.

Note that a RESTful endpoint does not need to be restricted to JSON data, it is perfectly ok to use a text/csv content type, and the actual implementation of this endpoint might be directly in the web server (if it can handle the authorization stuff correctly), so you don't need to go through some application REST framework although that would still be possible.

Answer 2

Is it a best practice to do this by API

No, one way to see it is, part of REST is having caches between client and server. Transferring file in one way means no caching is needed, so REST is not needed to do this.

You don't want to have a custom API for transferring file. There are many data integration tools that will just work with FTP, SFTP, FTPS, etc. to do this with little effort.