I am writing a program which polls a website every minute for updates on the weight of various types of widgets(A, B, C...)
Widget_A is returned I will:
- Download the accompanying timestamped PDF file
- Send it to AWS Textract, obtain the weight at from the document, and store it in my DB
- Compare it to the previous recorded weight for
Widget_A(if it exists in the DB, else return NA)
- Write out the % difference to my own webpage, as the topmost entry, pushing all previous entries down one line
Sometimes weights may be posted for multiple Widgets at once.
Each individual process is IO bound (sending to AWS Textract and waiting for the result).
Should I run a main loop which fires off these sub tasks using asyncio, as they arrive? Sometimes the Textract process might fail (never returns, or returns nonsense). How can I recover from this and keep the whole machine working indefinitely?