Note: All of this would be in AWS
What would you guys suggest for building something that:
- Takes in several different input file types (ex: csv, json, jsonl, xml, .gz, ...)
- That can be large (as of right now we know these can be multiple GBs)
- Maybe does some transformations / processing (requirements not quite as clear on this one yet)
- And sends all this data to our web services for further processing
We have a pretty simple node lambdas based design right now (at the bottom of the page), but I'm just looking for thoughts in terms of technologies we could leverage to make this less custom code, more flexible, and simpler that I can research into myself if you guys could help with any specific directions for me to explore.
Basic requirements known so far
- Be able to handle large files - at least multiple gigabytes
- This file would be in S3 - where this capability would be reading from
- In a reasonably performant manner - obviously everyone wants the multiple GB file to be processed and the data in our services within seconds, but that's not going to happen. AFAIK an hour or two may be OK here.
- This thing will NOT be writing to files or some DBs - it will be hitting our existing web services to send data to other parts of the platform
Possible but unknown requirements
- This thing may need to transform data
- And if possible, the transformation might need to be dynamic without an engineer going in and hard coding a transformation for each and every customer request. Ex: Customer A wants all 123s -> 456s But customer B wants all ABCs -> XYZs
- If possible, reduce custom code for the file ingestion / file transformation parts
I haven't dealt with files often in my career, and same for most of us here. I'd like to avoid running into a problem where someone not used to processing large files writes entirely custom code from scratch and unintentionally creates something not very maintainable / extensible / scalable / etc.
- Keep it simple. Only take on additional complexity if needed.
I'm not too familiar with the data tools out there like Spark, Hadoop, Airflow, ... From what I read so far (ex: first comment thread here) maybe some of these tools will be like trying to kill a fly with a nuclear bomb overkill.
If anyone has any thoughts around what would be good thing for me to poke around at, if there's something critical I missed, etc, happy to hear your thoughts.
- The big file ends up on S3
- S3 trigger to a node lambda
- This lambda breaks up the big file into a bunch of small files and uploads these small files to some other S3 bucket
- Yet another S3 trigger to another node lambda
- These lambdas would act on each small file, read the content line by line, and call the web services
Some challenges with this approach that I thought of would be:
- that's all custom code
- transformations...? if this ends up being a hard business requirement this might be a bit tough to introduce into this approach without yet even more customizations
- with this approach the second lambda would probably have a bunch running in parallel.
What if all these parallel calls to web services cause other problems further down the pipeline? We might have to do some reduces on the data, and that might mean yet even more custom code. How would it handle reducing multiple gigabytes worth of data? That's the one advantage I see with tools like Hadoop, Spark, etc who have figured these kinds of problems out that custom code wouldn't.