Airflow DAG vs custom worker to process queue of videos

Question

I need to process a queue of videos in a scalable way. Processing includes number of tasks that can depend on each other. Some tasks are computationally expensive some not (e.g. transcribe audio, clean up, summarize, extract features from transcription, etc.).

I think to create a DAG to define tasks dependencies and then execute the DAG for each task in a queue. Alternative approach is to create a custom worker that essentially will do the same: define the tasks which offload the work to other services.

Some airflow pros and cons.

Pros:

• airflow has all relevant concepts to explicitly defines tasks and their dependencies
• airflow has interface to track DAGs execution statuses
• people are familiar with airflow, but not with the custom worker code
• airflow gives a common solution if I need to run another jobs

Cons:

• not sure whether I can scale DAG executions
• airflow as a unit for scale looks like a wrong solution
• airflow is another system I have to manage
• airflow need a glue code that picks a task from the queue and starts DAG's execution
• custom code can be more flexible/customizable

I've not seen that Airflow used in a such way and a bit hesitant with such solution. What do you think?

Answer 1

From the documentation:

Apache Airflow is an open-source platform for developing, scheduling, and monitoring batch-oriented workflows.

The critical word in that was "batch" - I don't think you are going to be able to really leverage the UI as you would like. As the UI is very much focused on batches - finding a particular batch is not something it excels at - or to put it another way we typically ignore the UI unless a batch fails - which Airflow is good at highlighting.

Since Airflow is written in Python it is usually worth considering, if you have tasks written in python as they can be executed natively on the airflow cluster.

You can write bridging code to execute a task written in another language from the python code - for example in the worst case you can always exec() something else. Additionally Airflow can also be used to control other systems, for example if that other service exposes a REST interface.

However if you don't have a significant chuck of Python code to execute, Airflow is just another workflow engine and you should evaluate it against other engines as such.

Finally I would raise the question of: Is python the right language choice for computationally intensive jobs like video processing - the answer to that is going to be opinion based (hence off topic here), but it is something else you should consider.