1

our startup is working on an integration for various customers. All of the customers are required to provide their data in one unified way = we chose XML files because of legacy compatibility and ease to validate the files with XSD schemas.

The scenario should be that the customer provides their XML file which we will periodically (about 6 times a day) download and parse the data into our databases.

I am thinking that this could be managed by a single serverless function = simply parse the XML file with up to 1 million lines, compare with already existing records (if there are some updates) and if so, trigger an update event.

I am looking for a design or approach that would be the best match for this. Also, if there is a better way of what I just described, I am all ears for suggestions on how to improve the process.

Improve this question
3
  • 2
    This question got 4 close votes because of originally asking for 3rd party resources. Guys, instead of voting for close without a comment, you could have just stripped those buzzwords from the question (or leave a comment for the OP to strip them by themselves). That would have been way more constructive.
    – Doc Brown
    Commented Aug 7, 2018 at 8:40
  • @DocBrown OP's is referring to a lambda function declared and executed standalone somewhere (usually in a cloud platform). aws.amazon.com/lambda/?nc1=h_ls
    – Laiv
    Commented Aug 7, 2018 at 10:47
  • 1
    Sorry, by serverless function I mean AWS Lambda, Google Cloud Function or Azure Functions. I thought that the question was too much vendor specific.
    – Jan Richter
    Commented Aug 7, 2018 at 11:30

1 Answer 1

Reset to default
2

I think you will find that parsing a 1m line XML file potentially exceeds some of the limits on AWS Lambdas

https://docs.aws.amazon.com/lambda/latest/dg/limits.html

I would suggest the following

Multiple file upload methods to cope with different levels of tech expertise per customer.

  • FTP
  • Email
  • Website
  • Api

Async backend processing by a scaleable number of worker processes. These files could potentially take hours to process. I would keep it simple with windows services and a message queue. EC2 and SQS if you are using Amazon with a Backend database of your choice (RDS?)

Feed back validation errors to customer to request fix and re-upload. Again using multiple methods.

Allow customers to query the state of past and current Uploads via a website

Improve this answer
1
  • Any reason why would you go for Windows Services? I was thinking maybe do the parsing in Go or some other language running on a Linux server.
    – Jan Richter
    Commented Aug 7, 2018 at 10:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.