0

New to Apache Airflow and curious about how code and data are expected to be used across worker nodes in a multinode airflow setup.

When considering if ETL logic should be in the dags or in separate files called by operators in the dag, from here (https://www.astronomer.io/guides/dag-best-practices/) I see:

Try to treat the DAG file like a config file and leave all the heavy lifting for the hook and operator.

From this, my question is: 1) what is best way to sync the operator code across workers in a multinode airflow setup? And 2) given a multinode setup, are we expected to mount shared drives between all workers in order to store intermediate data (or does airflow have some other internal way to pass data between workers to complete dag tasks)?

1

Looking more into these questions, think I have some answers:

  1. From the article here (https://docs.bitnami.com/azure-templates/infrastructure/apache-airflow/configuration/sync-dags/) and discussions on the airflow mailing list, it appears that something like a git-sync on a cron schedule for dags and other required code across worker nodes is the norm.

  2. From here: https://gtoonstra.github.io/etl-with-airflow/principles.html

    An airflow operator would typically read from one system, create a temporary local file, then write that file to some destination system. What you should not and even cannot do is depend on temporary data (files, etc.) that is created by one task in other tasks downstream. The reason is that task instances of the same DAG can get executed on different workers and that local resource won’t be there.

    it seems clearer that we would need to mount shared drives between nodes (unless using something like AWS S3 or HDFS).

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.