I'm looking to put together a good architecture for developing and running Jupyter notebooks. We've been inspired by the Netflix model around the same but have little experience with Airflow and Papermill.

The examples and tutorials we have found all seem to focus on the component parts rather than bonding together usage.

I am trying to understand if either of these approaches is correct, possible and best practice or whether I should be doing/using something else.

Scenario 1
Airflow scheduler hits a relevant time and needs to run a DAG. IT uses something like ECS to create an instance based on a docker container that has papermill and any other dependencies relevant in it. Then the steps of the dag are executed within this instance/context.

Scenario 2
Airflow scheduler hits a relevant time and needs to run a DAG. The first steps of the dag are used to create an instance on EC2 that uses the docker container with papermill on it etc.

If anyone can direct me to any resources or give me a list of execution steps this would really help my learning journey!


Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Browse other questions tagged or ask your own question.