0

The client wants to put a CSV file on an FTP server, have it processed then have an error file put back in a different directory.

We are only a small company so we can only afford to support Java as the language.
We also don't have an FTP server so we are considering using Azure blob storage with a blob trigger and Function to act like it. The actual processing of the data is done in a service behind a REST API.

During prototyping we used Apache Camel to watch a directory, split the file into lines, convert it into JSON and upload each line (in JSON) to the REST API.

The Azure blob trigger will now manage the monitoring part so we can skip the Camel directory watcher.

Now that Azure functions are in the mix there are several options for this process.

  1. Put the whole Camel application in the Function to process the whole file in the existing way
  2. Have 2 Functions where the first does the splitting and the second uploads each line
  3. Have the Function upload the whole file to a new API endpoint and do all the splitting and process in the REST API

Which of these would be the most appropriate scenario for this use case and what would be the pattern for storing the error file back on the FTP server/blob storage?

NB: This would be staying in production for about 5-10 years (although it would probably be updated/changed during that time.
It doesn't need to be any more complex than the description as the processing of the data is not involved at this stage of the application.

It also doesn't need to be extendable as the features are well defined in advanced as it is replacing a very old system that hasn't really changed any functionality for a decade.

The max is likely to be 100k rows a month so it doesn't need to scale out either.

6
  • 1
    The three presented options are equivalent in functionality, but different in size and complexity. You've only really explained what functionality you need, and not your expectations of this application's lifetime in terms of complexity, size, and extensibility.
    – Flater
    Jul 7 at 15:37
  • 1
    Agree with @Flater, just pick what you think is best and go with it.
    – Dan Wilson
    Jul 7 at 17:35
  • I updated the question with details asked in the comments. The point of the question is to find the most appropriate pattern for the use case. If I was going to "pick what I think is best" I wouldn't have posted the question.
    – opticyclic
    Jul 7 at 17:53
  • I don't see any advantage in option 2. It seems to be additional complexity for absolutely zero gain. Choose option 1 or 3, depending on whether you feel more comfortable with having the complexity of splitting in a Function or your REST service. Jul 7 at 18:13
  • "We also don't have an FTP server" - does the client have their own server? Maybe they just need an internal app that will pick up the file from their internal server (that perhaps their colleagues have access to and use in their workflow), do something with the file, and put the output in a different dir someone else can access. I know this seems inelegant, but companies often have such workflows, and it's likely that introducing Azure blob storage and REST APIs and all this other stuff will not actually be helpful to them, or fit well within their established processes. Jul 7 at 20:33
2

Since you've already got a working solution, it's best to leverage what you've already built. You have a couple of approaches you can take:

  • Azure Blob trigger invokes an Azure Function, and whole application is in the azure function
    • Since your function is in Java, you will have slow start up times and that can add up in costs
  • Azure Blob trigger invokes an Azure Function written in Python, which calls an endpoint on your service.
    • Code to maintain can be small, and very little to configure
    • Python is commonly used for infrastructure and starts up quickly
    • Glue code is minimal.

If your Java service is built in Spring Boot, you at least have a single Jar file that contains all the dependencies. I like the second option a bit better, since it can integrate with Spring Boot Config Server.

0

You can always go with the AWS stack.

  • Use S3/Glacier for uploading/storing the CSV (Glacier will be slow compared to S3 but it is cheaper than S3.)
  • And for processing csv files, can always use lambda functions. (Costing of lambda function is based upon the, lambda invocation done)
1
  • We are using Azure and not AWS. We can't switch as we have Azure credits to pay for it.
    – opticyclic
    Jul 7 at 16:37

Not the answer you're looking for? Browse other questions tagged or ask your own question.