What Should An FTP to REST Process Look Like?

Question

I am writing a new application where the clients want to communicate using files over (S)FTP.

The first use case is that the client puts a file on the FTP server at end of day. I then process it and upload it line by line to my REST API.

The second use case is that there will be 2 or 3 drops intra-day for error processing.

At the moment I have a proof of concept that is just a main class that uses Apache Camel to gets a csv in a directory, reads it line by line, converts it, uploads each item to the REST API and shut down.

My reason for this idea was that it could be a batch process run at an agreed upon time at end of day and assume that the file was there or write (email?) an error if it was missing.

This wouldn't really work for the second use case though as I imagine that I should be watching the FTP directory constantly for new files.

Having a single batch process didn't seem to require much in the way of reliability and complexity.

Running it continuously seems like it might be more complicated and error prone and might require something like Spring Boot or a webserver or something.

What would be the model for this?

Answer 1

The main problem is what will you do if the process fail somewhere during the read and rest invocation. There are two approaches:

1. All or nothing. You read the whole file and try to upload it in one go. You don't need a problem resolution strategy because a file is either uploaded or not uploaded.

2. You upload it on chunks. I would suggest you to first read the lines and upload them in a queue or another vessel where you can in an easier way read the chunks and call your web service then you are back to 1) for each chunk.

If you are using Kafka f.ex. there may be available implementations of source and sync.

Usage if Kafka is definitely non mandatory any persistent store with Queue or Stack semantic will do the trick.

With regards of scheduling you may not need Spring boot, although spring boot async is not that bad as combination. On operating system level you can configure a cron job.