0

How do "you" handle runtime incidents (say, a null-pointer exception, or an API input data validation failure) so that you (a) don't miss any, and (b) aren't spammed by duplicates? Are you doing log file introspection?

We're currently NOT using Jira for managing bug reports. We are going to want to do that.

In reality, if a runtime exception occurs, it's going to re-occur numerous times before we have a fix deployed to production, and we don't want to be notified 187 separate times about what is essentially a known issue.

What we've been doing (for about a decade) is:

  • When a runtime exception handler can't resolve an incident programmatically, it will send an email (with metadata, stack trace, etc.) to a mailbox. (The email also serves as a one-way pipe out of a tightly regulated production environment.)
  • The mailbox is monitored by a separate system that does some regex matching to generate standardized "bug reports" from these emails.
  • A recurring task will then classify new bug reports in order to identify and assign the relevant project (software product) and the owning development team. The classifier will also identify re-occurrences of known issues and attach them as "children" to the original bug report for that issue. We can also tweak the parameters for how a "parent" bug report will classify re-occurrences.
  • Individual bug reports can then be pulled into our sprints as tasks, but a bug report remains "open" (still catching re-occurrences) until it is marked as deployed.

This is a very effective system to "squelch" the stream of bug reports, allowing us to focus on actually unique issues.

...but all of this is outside of Jira, and Headquarters really wants us to "embrace Jira" so we need to think about how to retain the value of this triage automation.

(For clarification, I'm attaching a sample screen shot of our triage automation tool.)

bug-report-auto-triage

  • 1
    I won't post this as an answer since it's not what you asked for, but why not leverage JIRA's REST API to simply integrate your existing process with it? – Jan Dorniak Sep 27 at 10:10
  • @JanDorniak, that is indeed my plan -- when I have sufficient assurance we are not re-inventing the wheel. :) – KlaymenDK Sep 27 at 10:13
  • Why do you need to change anything, if your current system works? What does "embrace Jira" actually mean, and why do people want you to do it? To put it another way, given that a change will mean work, what's the value you get in return? – jonrsharpe Sep 27 at 10:15
  • @jonrsharpe, the idea is to rely on off-the-shelf products where we can, since we are not in the business of building bug-reporting tools (we just need them). Hence, I need to show that the work is indeed necessary because no better solution is available. Also, Headquarters already views Jira as the Golden Hammer and wants everything that can be done in Jira, done in Jira. – KlaymenDK Sep 27 at 10:17
  • 1
    @jonrsharpe: The reasoning also applies to the situation "if it breaks, can we offload the work of fixing it to a supplier or do we need to wast our own resources on it." – Bart van Ingen Schenau Sep 27 at 11:01
1

You need a logging stack. ELK, app insights, splunk etc

So your app just logs the error, to a file, syslog,api call whatever and moves on. No sending emails, just fire and forget.

These logs are then collected and combined into a timeseries database. ie logstash monitors the file, app insights saves the api call, splunk.. splunks

The time series database is then queried on a schedule to trigger alerts. For example, how many errors in the last 5min? if greater than 10 the trigger an action. You can get inventive with your rules, applying machine learning, floating thresholds etc

Now the action might be "send an email" but in a more advanced setting it will be "create an incident"

Creating a incident means that you can collect multiple alerts together. So if the same alert triggers next time it runs, that would automatically be rolled into the incident, but also other related alerts can be rolled up according to various rules that you setup.

An incident can trigger all sorts of actions. email staff, auto remediate, open tickets, call other apis etc etc. plus you can add information to it. "bob: turned server on and off. no effect", "jim: after reviewing this case, in retrospect it turned out to be a bug"

At the end of the quarter you can query your incidents and get a whole load of information about what was affected, what actions were taken, how long it went on for etc etc

  • "you can collect multiple alerts together" - but would you have to do this collection manually, or could you set the system up to auto recognise things it's seen before? – KlaymenDK Sep 27 at 18:37
  • both, you set the rules and relationships up so that some event are grouped, but you can also intervene manually – Ewan Sep 27 at 18:39
0

We're using a hash over non-variable parts of the call stack for exactly this purpose. In our case, we automatically place tickets into Redmine, adding a link to the previous occurrence of an error with the same hash. Works reasonably well.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.