I have been working on a cloud-based background worker service, and regrettably it is almost "blind". What I mean by this is it is incredibly difficult to get any insight into what is going on, aside from the tedious process of reading through noisy logs.
Some of the common things I would like to know are:
- is it running?
- if/when did it crash
- the result of business logic calculations
- data used during failed conditional checks
As I mentioned I can pull this information from the logs, but it is noisy from dependencies and other tools logging at the same levels. So I would like to prioritize key performance data from other log details.
One way I could obviously resolve this is by creating a separate custom dashboard UI. This is do-able with the talent on our team, but the time and effort required may not be justifiable.
What I'm looking for are good strategies for getting insight into a background service that are better than looking through raw text logs, while also being less time-consuming to build than a custom UI?