0

I'm new to Airflow and I'm working on a proof of concept. The project is fairly simple... every day some 10,000 JSON files are loaded onto a folder on AWS S3. I have to get each one of them, parse them and then load them into a database.

It seems like this just requires on PythonOperator? This is my idea:

  1. Check the S3 folder for files.
  2. Read one file and parse it. This will generate about 5 flattened records per JSON file. I think I can read the file without explicitly downloading it somewhere.
  3. Insert these 5 records into the database
  4. Delete/archive the file
  5. Repeat steps 1-4 with each file until there are no more files in the folder

I come from a SSIS background and these would all be different tasks so I'm curious if there's a better way in Airflow. I noticed there's a MSSQL operator but I don't think that's much cleaner anyway.

Gabe

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.