I need to process a queue of videos in a scalable way. Processing includes number of tasks that can depend on each other. Some tasks are computationally expensive some not (e.g. transcribe audio, clean up, summarize, extract features from transcription, etc.).

I think to create a DAG to define tasks dependencies and then execute the DAG for each task in a queue. Alternative approach is to create a custom worker that essentially will do the same: define the tasks which offload the work to other services.

Some airflow pros and cons.


  • airflow has all relevant concepts to explicitly defines tasks and their dependencies
  • airflow has interface to track DAGs execution statuses
  • people are familiar with airflow, but not with the custom worker code
  • airflow gives a common solution if I need to run another jobs


  • not sure whether I can scale DAG executions
  • airflow as a unit for scale looks like a wrong solution
  • airflow is another system I have to manage
  • airflow need a glue code that picks a task from the queue and starts DAG's execution
  • custom code can be more flexible/customizable

I've not seen that Airflow used in a such way and a bit hesitant with such solution. What do you think?

1 Answer 1


From the documentation:

Apache Airflow is an open-source platform for developing, scheduling, and monitoring batch-oriented workflows.

The critical word in that was "batch" - I don't think you are going to be able to really leverage the UI as you would like. As the UI is very much focused on batches - finding a particular batch is not something it excels at - or to put it another way we typically ignore the UI unless a batch fails - which Airflow is good at highlighting.

Since Airflow is written in Python it is usually worth considering, if you have tasks written in python as they can be executed natively on the airflow cluster.

You can write bridging code to execute a task written in another language from the python code - for example in the worst case you can always exec() something else. Additionally Airflow can also be used to control other systems, for example if that other service exposes a REST interface.

However if you don't have a significant chuck of Python code to execute, Airflow is just another workflow engine and you should evaluate it against other engines as such.

Finally I would raise the question of: Is python the right language choice for computationally intensive jobs like video processing - the answer to that is going to be opinion based (hence off topic here), but it is something else you should consider.

  • Thanks for the answer! What lures me to Airflow are its task dependency concepts, DAG run reports, and UI. Other than that, it doesn't seem like an ideal choice. Video processing is on-demand execution, not a batch job on a schedule. Python is also not that important. All compute-intensive tasks will be outsourced to other services, e.g., GCP Speech-to-Text. If I need to process smaller tasks in the future, spinning up an Airflow DAG for each task will probably become a performance bottleneck. Also, it's another system to manage.
    – vimi
    Sep 25 at 21:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.