1

I have to upload a TSV file to an S3 bucket, the content is form submission data. Another service will consume and do things with this data as soon as possible. The service would like one large file that is appended with each new form submission.

My simple approach is submit --> download data --> append --> upload.

The problem being two people submitting at the same time and data being lost.

Is there a sane and straight forward way of avoiding this? Batch processing of single files is out of the question as the data needs to be used straight away. I was considering inserting into a SQL Server table, and then exporting the TSV on each new row via a trigger. Apparently this is a bad idea and a service broker should be used instead but I don't know anything about this.

The approach needs to be .NET C# and I cannot avoid needing to upload a TSV file of form data. Perhaps there is something I can do within .NET itself?

Any suggestions are welcome, thanks.

|improve this question|||||
  • Sounds like you need to implement a lock of some sort? – Ant P Jul 26 '19 at 9:57
  • can you point me in the right direction? Search terms / documentation to look at etc.? All very new territory for me. Thanks. – monkeySeeMonkeyDo Jul 26 '19 at 10:39
  • use a database? – Ewan Jul 26 '19 at 11:29
  • 1
    Don't create a big file. Instead, maintain a record of the individual uploads, and when asked to deliver the big file, concatenate the pieces on the fly. – BobDalgleish Jul 26 '19 at 14:03
  • this is the approach I wanted and planned originally, but the consumer wants a large file right away. – monkeySeeMonkeyDo Jul 26 '19 at 14:05
2

Make a .net web service that receives the data submitted by the form.

Add that data to a database, using transactions to maintain consistency

After adding the data to the Database call a second FileExport

FileExport reads the database, generates the file and writes it to the s3 bucket.

FileExport also buckets incoming requests. So if you get 50 updates in a second, it doesn't write the file 50 times. Have a user configurable setting for the bucket size.

Now the database ensures that you will never overwrite other peoples updates and the FileExport service ensures that the file is upto date with the latest changes.

Alternatively, instead of having FileExport and an s3 file, have consumers of the file call the Export service directly. Which can then generate a file from the database with the latest data at the time of their call.

|improve this answer|||||
  • thanks, this sound like what I need. Unfortunately the consumer requires the file to be uploaded to S3, so them calling the web service is out of the window. I'll take a look at this approach, it seems straight forward enough. – monkeySeeMonkeyDo Jul 26 '19 at 12:09
  • what's the best thing to look at / google for bucketing / pooling incoming requests? – monkeySeeMonkeyDo Jul 26 '19 at 14:06
  • hmm i guess it is a tricky one. a simple solution is to have a scheduled task check the db every x seconds and remember the last update it sent out – Ewan Jul 26 '19 at 17:31
  • if you use a message queue, you can pull requests and wait to see if there are more before processing – Ewan Jul 26 '19 at 17:32
1

I think it is best not to append the files while uploading. You could upload the different files. Then you could have a lambda trigger on new files in S3 which does the concatenation.

But in any case there will be a short moment where no file is available while the concatenated file is being uploaded.

|improve this answer|||||

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.