The client wants to put a CSV file on an FTP server, have it processed then have an error file put back in a different directory.
We are only a small company so we can only afford to support Java as the language.
We also don't have an FTP server so we are considering using Azure blob storage with a blob trigger and Function to act like it. The actual processing of the data is done in a service behind a REST API.
During prototyping we used Apache Camel to watch a directory, split the file into lines, convert it into JSON and upload each line (in JSON) to the REST API.
The Azure blob trigger will now manage the monitoring part so we can skip the Camel directory watcher.
Now that Azure functions are in the mix there are several options for this process.
- Put the whole Camel application in the Function to process the whole file in the existing way
- Have 2 Functions where the first does the splitting and the second uploads each line
- Have the Function upload the whole file to a new API endpoint and do all the splitting and process in the REST API
Which of these would be the most appropriate scenario for this use case and what would be the pattern for storing the error file back on the FTP server/blob storage?
NB: This would be staying in production for about 5-10 years (although it would probably be updated/changed during that time.
It doesn't need to be any more complex than the description as the processing of the data is not involved at this stage of the application.
It also doesn't need to be extendable as the features are well defined in advanced as it is replacing a very old system that hasn't really changed any functionality for a decade.
The max is likely to be 100k rows a month so it doesn't need to scale out either.