Note: All of this would be in AWS

Hi everyone,

What would you guys suggest for building something that:

  • Takes in several different input file types (ex: csv, json, jsonl, xml, .gz, ...)
  • That can be large (as of right now we know these can be multiple GBs)
  • Maybe does some transformations / processing (requirements not quite as clear on this one yet)
  • And sends all this data to our web services for further processing

We have a pretty simple node lambdas based design right now (at the bottom of the page), but I'm just looking for thoughts in terms of technologies we could leverage to make this less custom code, more flexible, and simpler that I can research into myself if you guys could help with any specific directions for me to explore.

Basic requirements known so far

  • Be able to handle large files - at least multiple gigabytes
  • This file would be in S3 - where this capability would be reading from
  • In a reasonably performant manner - obviously everyone wants the multiple GB file to be processed and the data in our services within seconds, but that's not going to happen. AFAIK an hour or two may be OK here.
  • This thing will NOT be writing to files or some DBs - it will be hitting our existing web services to send data to other parts of the platform

Possible but unknown requirements

  • This thing may need to transform data
  • And if possible, the transformation might need to be dynamic without an engineer going in and hard coding a transformation for each and every customer request. Ex: Customer A wants all 123s -> 456s But customer B wants all ABCs -> XYZs

My goals

  • If possible, reduce custom code for the file ingestion / file transformation parts

I haven't dealt with files often in my career, and same for most of us here. I'd like to avoid running into a problem where someone not used to processing large files writes entirely custom code from scratch and unintentionally creates something not very maintainable / extensible / scalable / etc.

  • Keep it simple. Only take on additional complexity if needed.

I'm not too familiar with the data tools out there like Spark, Hadoop, Airflow, ... From what I read so far (ex: first comment thread here) maybe some of these tools will be like trying to kill a fly with a nuclear bomb overkill.

If anyone has any thoughts around what would be good thing for me to poke around at, if there's something critical I missed, etc, happy to hear your thoughts.

Current design:

  • The big file ends up on S3
  • S3 trigger to a node lambda
  • This lambda breaks up the big file into a bunch of small files and uploads these small files to some other S3 bucket
  • Yet another S3 trigger to another node lambda
  • These lambdas would act on each small file, read the content line by line, and call the web services

Some challenges with this approach that I thought of would be:

  1. that's all custom code
  2. transformations...? if this ends up being a hard business requirement this might be a bit tough to introduce into this approach without yet even more customizations
  3. with this approach the second lambda would probably have a bunch running in parallel.

What if all these parallel calls to web services cause other problems further down the pipeline? We might have to do some reduces on the data, and that might mean yet even more custom code. How would it handle reducing multiple gigabytes worth of data? That's the one advantage I see with tools like Hadoop, Spark, etc who have figured these kinds of problems out that custom code wouldn't.

  • Is there anything which will guarantee the quality and validity of the files being landed into your S3 bucket? If not, have you considered a way to identify poor quality files (e.g. valid format but nonsense or incomplete data), or even invalid files, and then what you would do with them? IMO it's best to deal with this sort of thing up-front as early as possible before you attempt to run any kind of transformations. (i.e. quality/validation checks being a separate concern and taking place before transformation) Jan 31, 2021 at 0:25
  • Hi @BenCottrell - Hmm, I'm not sure that there's a 100% of the way to validate everything up front, but that is the general idea we had as well. High level thoughts so far: 1. A lambda to break up the multi GB file to small chunks (say 1MB each) 2. Lambdas to act on these small chunks. Validate and then send the data to backend web services. The question then was where should transformation go if we end up having to do them - up front before the file upload as a separate thing, as part of lambas in #2, etc I've been researching and leaning towards Python pandas for this
    – user384015
    Feb 1, 2021 at 14:51


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy