I am writing a program which polls a website every minute for updates on the weight of various types of widgets(A, B, C...)

If Widget_A is returned I will:

  1. Download the accompanying timestamped PDF file
  2. Send it to AWS Textract, obtain the weight at from the document, and store it in my DB
  3. Compare it to the previous recorded weight for Widget_A (if it exists in the DB, else return NA)
  4. Write out the % difference to my own webpage, as the topmost entry, pushing all previous entries down one line

Sometimes weights may be posted for multiple Widgets at once.

Each individual process is IO bound (sending to AWS Textract and waiting for the result).

Should I run a main loop which fires off these sub tasks using asyncio, as they arrive? Sometimes the Textract process might fail (never returns, or returns nonsense). How can I recover from this and keep the whole machine working indefinitely?

Many thanks!

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.