I know there are a number of articles only and tutorials that show using RedShift COPY to get a CSV file/content into a redshift database. My question is on implementation.
My thought is to place a lambda function on the S3 bucket that is triggered when a file is added to the bucket. The lambda would use aws-sdk and node-redshift to get the CSV file and then copy that into the redshift table.
file drop -> S3 -> lambda/node.js -> redShift ? or file drop -> S3 -> lambda/node.js -> DynamodDB -> redshift
Does this sound feasible? Other write-ups I have read suggest taking the data and sending it to DynamoDB and then copy from there to Redshift. I've also seen a server less implementation but this seems like overkill...
These files are very light and I don't think that I need an overly complex solution. Any recommendations on a pattern/implementation to follow would be appreciated.