I need to process a queue of videos in a scalable way. Processing includes number of tasks that can depend on each other. Some tasks are computationally expensive some not (e.g. transcribe audio, clean up, summarize, extract features from transcription, etc.).
I think to create a DAG to define tasks dependencies and then execute the DAG for each task in a queue. Alternative approach is to create a custom worker that essentially will do the same: define the tasks which offload the work to other services.
Some airflow pros and cons.
- airflow has all relevant concepts to explicitly defines tasks and their dependencies
- airflow has interface to track DAGs execution statuses
- people are familiar with airflow, but not with the custom worker code
- airflow gives a common solution if I need to run another jobs
- not sure whether I can scale DAG executions
- airflow as a unit for scale looks like a wrong solution
- airflow is another system I have to manage
- airflow need a glue code that picks a task from the queue and starts DAG's execution
- custom code can be more flexible/customizable
I've not seen that Airflow used in a such way and a bit hesitant with such solution. What do you think?