Best technique for pushing many records and files to a system: WEB API or FTP

Question

I have a website where users can upload manuals (PDF documents) for their products. Basically, they first add the product to the system and then add 1 or more manuals to that product. A manual can have multiple versions and all versions should be available in the system. In addition, manuals can contain multiple languages. That's briefly it.

This approach has always worked well, but lately we've been having customers with tons of products/manuals and already owning a system containing that data. They're not interested in manually adding all that data, they want a system to push their products and manuals to our website automatically, including updates and deletions. So it's basically a 1 direction operation from their system to our website.

I have built WEB APIs before, but it was limited and only consisted of exchanging record data, not files and certainly not a collection of data/files. So my first question would be, can I post a collection of data with a collection of files associated with that data to a WEB API?

E.g.:

• Product A
  • Manual1.PDF
  • Manual2.PDF
• Product B
  • Manual3.PDF
• Product C
  • Manual4.PDF
  • Manual5.PDF
  • Manual6.PDF

Can I post this data and still associate Maual1.PDF with Product A on the server? Should I avoid trying to send as much associated data crammed into 1 POST? Should I force the client to only send 1 product with its manuals one by one, to avoid running into long HTTP requests? Any tips in general how I should be doing this?

In the past, we would build synchronization systems using FTP to upload all the PDF documents and 1 XML file that contained all the data and a reference to its associated PDF document. A job on the server would look for an uploaded XML file and process it when found. While this does work, I feel like this is an outdated solution and not really a professional approach.

Edit: Perhaps I wasn't clear enough in my description, but what I'm looking for is an interface that I will build on top of my website, which my customers can integrate in their software to push data automatically to my website whenever they make a change in their system. I want to build this interface in the ASP.NET WEB API technology, but I'm not experienced enough in that technology to determine if it is capable of what I described above: accepting files and nested data refering to those files in 1 post. And that maybe, a solution where the customer's system automatically uploads the files along with an index file through FTP would be better suited? Or a different technology?

Answer 1

Have you looked into existing so-called "Content Management Systems" (CMS)?

In case you want to roll your own system, though, a REST endpoint with a single page web app as the frontend with a user interface for selecting the files to be uploaded running in a user's browser sounds like a reasonable approach. Especially as these are technologies for which it should be fairly easy to find somebody experiences to implement the system.

An open FTP endpoint sounds scary from a security point of view. If anything, please go for SCP or SFTP. The implementation and managagement of such a system would probably be harder than the REST + browser approach, though, because systems based of SCP / SFTP / FTP are much less common, and so are the necessary skills and experience in the field for quick and easy and robust implementation.

Answer 2

In isolation assuming you had all solutions ready for a large amount of files FTP is the better solution, even if its not the new hotness. FTP is a good solution for moving lots of files from point A to B in an efficient manner. Another option would be to allow VPN access to a file server in your environment and let clients upload documents that way. The downside for a FTP solution is the it will likely require action from your clients network admins to open connections to it, where web API can piggyback on open http ports.

If neither solution exists and they need to be developed, then the best solution is one that will continue to provide value for other clients. In most cases a simple crud API for manuals would be sufficient for clients that have few manuals. If you have a lot of clients that have lots of manuals and they make mass changes it would be beneficial to have a system better designed to work with files in bulk.