I am designing the API architecture for my client:
My client actually exchange huge files (csv) with his Partner with MFT ( SFTP )
- **Partner ==> MFT ==> Client **
The requirement is to move to API REST exchange mode to replace MFT ( csv file)
- **Partner ==> API ==> Client consume GET API **
For example :
The partner of my client send every day 200 MB of text data ( csv file 200 000 lines)
Actually my client receives this by MFT ,he asks me if it is best practise to replace this by API REST CALL to transfer large data ( order of 200MB)
I know the best practice for a normal payload is in order of 1 MB ( json payload) for one call HTTP, so I will be doing pagination (200 Pages) and handle errors and so on.
- Is it a best practice to do this by API?
- Are there detailed / practical arguments for this ( why we should move to API in this case , my client always asks me this )?
- Are there techniques to reduce pagination while transfer data 200 MB ( I estimated 200 pages and 200 http call to get all data ) with API while keeping performance ?