Parsing a JSON file from S3 using Airflow

Question

I'm new to Airflow and I'm working on a proof of concept. The project is fairly simple... every day some 10,000 JSON files are loaded onto a folder on AWS S3. I have to get each one of them, parse them and then load them into a database.

It seems like this just requires on PythonOperator? This is my idea:

1. Check the S3 folder for files.
2. Read one file and parse it. This will generate about 5 flattened records per JSON file. I think I can read the file without explicitly downloading it somewhere.
3. Insert these 5 records into the database
4. Delete/archive the file
5. Repeat steps 1-4 with each file until there are no more files in the folder

I come from a SSIS background and these would all be different tasks so I'm curious if there's a better way in Airflow. I noticed there's a MSSQL operator but I don't think that's much cleaner anyway.

Gabe