Best practice for sharing code and data between airflow worker nodes?

Question

New to Apache Airflow and curious about how code and data are expected to be used across worker nodes in a multinode airflow setup.

When considering if ETL logic should be in the dags or in separate files called by operators in the dag, from here (https://www.astronomer.io/guides/dag-best-practices/) I see:

Try to treat the DAG file like a config file and leave all the heavy lifting for the hook and operator.

From this, my question is: 1) what is best way to sync the operator code across workers in a multinode airflow setup? And 2) given a multinode setup, are we expected to mount shared drives between all workers in order to store intermediate data (or does airflow have some other internal way to pass data between workers to complete dag tasks)?

Answer 1

Looking more into these questions, think I have some answers:

1. From the article here (https://docs.bitnami.com/azure-templates/infrastructure/apache-airflow/configuration/sync-dags/) and discussions on the airflow mailing list, it appears that something like a git-sync on a cron schedule for dags and other required code across worker nodes is the norm.

2. From here: https://gtoonstra.github.io/etl-with-airflow/principles.html

   An airflow operator would typically read from one system, create a temporary local file, then write that file to some destination system. What you should not and even cannot do is depend on temporary data (files, etc.) that is created by one task in other tasks downstream. The reason is that task instances of the same DAG can get executed on different workers and that local resource won’t be there.

   it seems clearer that we would need to mount shared drives between nodes (unless using something like AWS S3 or HDFS).