Dependency resolution of tasks which have crontab information

Question

I want to illustrate my problem statement with a use case. I am building a trading system that acts as a container for deploying trading strategies. The trading strategy comprises dependent tasks. Let's suppose there is task T1 and T2. T1 runs at the market open (the core of the strategy), and T2 runs after the market closes (kind of like an analysis task). The two requirements I have are:

1. I want T2 to run after T1 and after the market closes. So I can leave system resources for other priority tasks that run during market hours (Note: T1 can complete way before the market closes).

2. I want the next run of T1 (not the first) to wait till the previous run of T2 to achieve something like a feedback loop.

The way I see it, to accomplish the first requirement, I need to associate the cron information to the tasks along with the dependencies. And to accomplish the second requirement, introduce something like a future dependency (T2 depends on T1, but T1 future depends on T2).

Now, I have used some workflow schedulers like Apache Airflow. But it has cron information at the DAG level, not at the task level. (2.) can be accomplished by disabling parallel runs. Mine will be a single big workflow with each task running according to its crontab schedule.

I wonder if an algorithm/library exists that does the dependency-resolution and cycle-detection for the use case I mentioned above. Looking forward to hearing from Graph(ites)

Please feel free to correct my thinking if I am complicating it :)

Answer 1

I had a very similar use-case years ago.

The simplest way of thinking here is a “system throughput”. Eventually you want both tasks running daily, and technically you have to keep them happening one by one. Let’s ask a few questions that should help here:

1. Can we skip the day?
2. Can we run a job on next day?
3. Running job during next business day would be as simple as running it today, or require dirty hacks?

This is not only about keeping cycle, this is also about your ability to execute job with correct inputs and complexity of execution process.

I end up with understanding that no matter what, the job need to be done within the same day, i.e, running today’s job tomorrow is a big problem.

As a consequence, the monitored sequence is on day level. If the job fails I have to stop batch process and wait for a fix. If I run the current day's work on a future day (historically), that's a totally different process that must acknowledge datetime change in my use case.

Answer 2

cron isnt a great way to schedule stuff when you have complex dependencies and logic.

Even complex 3rd party workflow solutions have limitations like the one you describe with airflow.

If you have complex logic like what you describe, which presumably also has al sorts of error conditions and edge cases, its best to simply program the flow in another worker T3.

You can then hold onto all the information you require for your logic, the history of runs, how long the run takes, whether T2 is still in progress etc etc and implement whatever logic you require.