I have to upload a TSV file to an S3 bucket, the content is form submission data. Another service will consume and do things with this data as soon as possible. The service would like one large file that is appended with each new form submission.
My simple approach is submit --> download data --> append --> upload.
The problem being two people submitting at the same time and data being lost.
Is there a sane and straight forward way of avoiding this? Batch processing of single files is out of the question as the data needs to be used straight away. I was considering inserting into a SQL Server table, and then exporting the TSV on each new row via a trigger. Apparently this is a bad idea and a service broker should be used instead but I don't know anything about this.
The approach needs to be .NET C# and I cannot avoid needing to upload a TSV file of form data. Perhaps there is something I can do within .NET itself?
Any suggestions are welcome, thanks.