I have an application that is receiving a live stream of files, I am using active MQ for listening to the file arrival in s3. As soon as the file arrives in s3 my application downloads the file and starts processing for it.

Now, I require to pause the processing for certain types of files (*Xyz*.txt, pause window is 8 AM UTC to 9 AM UTC) and resume processing whenever the pause window is over. I know the pause time window for each file. For instance, If a file with the pattern as *Xyz*.csv is received between 8 AM UTC to 9 AM UTC then the application will not process that file during the pause window, but after 9 AM UTC, it will start processing that file.

What are the ways through which I can resume file processing? I was exploring the Kubernetes cronjob type application, which will poll my core application using some API in a 5 min window and if any files pause window is over it will pick that file and start processing it.

The one major problem I see with this approach is the cronjob application is polling every 5 min, which is making several calls to my core application.

Is there any better way to solve this problem?

Please pardon me if my question is very naive.

  • If you detect a file matching the pattern during the pause window, can you push that message for the file back on the queue? Sep 11 '21 at 13:01

I'd push files onto a "deferred" queue during their pause window. After the pause window ends, you can move all files from the "deferred" queue to the main queue. If the pause windows always end on the full hour, an hourly job should be sufficient.

  • it sounds good although I would still need a scheduler to move the files from the deferred queue to the main queue. However, if I design it with a pause window ending on the full hour, it would significantly reduce the scheduler overhead. Thanks
    – DDM
    Sep 12 '21 at 5:21
  • You didn't mention why those pause windows are needed. Is there some process running which shouldn't be disturbed by new files? If yes, you might use the completion of this process as the trigger. However, you'll need some state variable to note that the pause window is over if it's not on a fixed time. Perhaps you have an XY problem here. Sep 12 '21 at 5:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.