1

I have been working on a cloud-based background worker service, and regrettably it is almost "blind". What I mean by this is it is incredibly difficult to get any insight into what is going on, aside from the tedious process of reading through noisy logs.

Some of the common things I would like to know are:

  • is it running?
  • if/when did it crash
  • the result of business logic calculations
  • data used during failed conditional checks

As I mentioned I can pull this information from the logs, but it is noisy from dependencies and other tools logging at the same levels. So I would like to prioritize key performance data from other log details.

One way I could obviously resolve this is by creating a separate custom dashboard UI. This is do-able with the talent on our team, but the time and effort required may not be justifiable.

What I'm looking for are good strategies for getting insight into a background service that are better than looking through raw text logs, while also being less time-consuming to build than a custom UI?

  • 1
    What about creating a hybrid solution, a tool to parse the relevant info from the logs and sending a report of some sort? – PmanAce Nov 18 '16 at 19:41
1

If you already have the log files and they contain all the data you need but are just too hard to go through, you might use the ELK stack to extract the data you want.

The ELK stack is short for:

  • ElasticSearch
  • LogStash
  • Kibana

enter image description here

Installing and configuring the tools might be faster than building your own dashboard.

If you think this setup is too much and just as time consuming as building your own dashboard, then user PmanAce provided you with a good solution in his comment.

  • This will still require a way to categorize entries. Why not make another step and just add a dedicated prefix to service's messages? This would allow using simpler tools. – Basilevs Nov 20 '16 at 11:16
  • @Basilevs: I added an answer for "the tedious process of reading through noisy logs" with tools that allow you to draw multiple conclusions by looking at the same data in multiple ways. But I agree with you. Use whatever gets the job done. – Bogdan Nov 20 '16 at 16:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.