I'm new to Airflow and I'm working on a proof of concept. The project is fairly simple... every day some 10,000 JSON files are loaded onto a folder on AWS S3. I have to get each one of them, parse them and then load them into a database.
It seems like this just requires on PythonOperator? This is my idea:
- Check the S3 folder for files.
- Read one file and parse it. This will generate about 5 flattened records per JSON file. I think I can read the file without explicitly downloading it somewhere.
- Insert these 5 records into the database
- Delete/archive the file
- Repeat steps 1-4 with each file until there are no more files in the folder
I come from a SSIS background and these would all be different tasks so I'm curious if there's a better way in Airflow. I noticed there's a MSSQL operator but I don't think that's much cleaner anyway.