0

We are writing a service that can parse a CSV file containing over 1M+ rows. Each row needs insertion (or update) data of a DB record. We are currently using DynamoDB.

What AWS services are best suited for this? We are considering lambda and queue system.


Solution we are looking at.

  1. API: API receives csv file. The file is streamed and chunks are pushed to a queue.
  2. QUEUE: The queue holds chunks waiting to be processed.
  3. PROCESSOR: The chunk processor goes through the queue and inserts records to dynamo.

enter image description here


Challenges:

  • How would you trigger an end of processing notification (to indicate the last chunk for that request was completed).
  • How would you handle partial failure and rollback if using this approach?
1
  • I guess an AWS forum like repost.aws would be the better place to ask this question.
    – Doc Brown
    Apr 28 at 20:10

0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.