2

We do have a micro-services architecture with a team assigned to each micro-service considering it as a product on itself.

The "real product" is a front-end that uses multiple micro-services to offer functionality to the end-user (no micro front-ends), that means that all the bugs are reported at the front-end level as that's what the end-user experiences.

One normal flow would be:

  1. The front explores what is happening and sees a 500 returned by service A. He sends the bug to the team of service A.
  2. The service A team explores what happens and sees that when service A authenticates against service B he is getting a 401 with valid credentials. He sends the bug to the team of service B.
  3. The service B team explores why the 401 error and discovers that the service C returned an empty list of valid users. The bug is assigned to service C.
  4. The service C team detects a bug in their code and fixes it.

This means the following:

  1. All the bugs are reported to the front team and that means the front team is actually doing the triage of every single bug and supporting an extra workload.
  2. The bugs are reassigned between the teams in a "cascade" flow until the issue is detected increasing the time required to solve it due to the different delays between something gets assigned and investigated.
  3. When there is no clear root cause as the bug happens due to complex interactions is difficult to get someone leading the organization of the diagnostics as there is no clear owner.

How do you manage such situations with a similar organization?

3
  • While not an answer to your question, it's a clear indication that your microservices aren't autonomous enough. If a microservice gets a request, it should be able to handle the request without calling other microservices.
    – Rik D
    Nov 24, 2020 at 9:06
  • I am not sure if it is mandatory to have fully autonomous microservices as communications through message busses are quite common. Nov 24, 2020 at 11:00
  • That's true, in that sense no microservice can be fully autonomous; they always depend on some form of communication. With a message bus solution it's still possible that service A didn't send the event, or service B didn't handle the event correctly. I think loosely coupled systems are preferable over systems that communicate over HTTP API's, but for production bug tracing in the scenario you described, there's probably not much difference.
    – Rik D
    Nov 24, 2020 at 12:38

4 Answers 4

4

When your application is layered, with different layers being maintained by different teams, it is unavoidabe that bugs are passed around between these teams until they reach the team responsible for the failing component.

Slicing your products vertically helps mitigate this, as one team may be able to track down a bug through front end, back end, possibly some auxiliary microservice. You might want to reconsider your approach of one-tem-per-microservice as it hampers agility. When services are shared between products (such as traditional databases, or authorization services) you should at least establish a good working contact between the teams so they can talk with each other to identify and solve problems instead of just passing tickets around.

Your third point (no clear root cause) is fairly common when (micro)services and their clients are developed by different teams. Often the interfaces between services and clients are not rigorously specified, causing misinterpretations and communication failures. If there's no clear owner you still have stakeholders (the teams responsible for provider and consumer of such a service) who need to get together and get a consensus about the actual interface definition, and normally that leads to one side being required to change their code to adhere to the common understanding of the interface.

1
  • Thanks for the answer, I understand your point regarding issues and coordination during the build phase, we have that part reasonably well covered. I was referring specifically to production issues, I will add that to the question, anyway, your answer still adds value. Nov 24, 2020 at 11:18
3

Although Hans-Martin Mosner's answer hits most of the key points, there is another aspect to consider: observability. The ability to trace web requests through the system, viewing performance, requests and responses, and associated log messages can help a team to understand where the error occurred and streamline the triage process.

With your current structure, a triage team that contains representatives from each of the other teams may also help. Observability would support this triage team's ability to work. Although I would consider alternative team structures, like end-to-end ownership of features or functions across all services rather than ownership of individual services.

2
  • I think sharing features you end up having several teams with a shared codebase and diluting ownership. Defining clear boundaries on services as products that could be sold as third party services and assigning them to a team makes easier to structure the organization and to provide ownership to the solution. Is there a common approach of the community regarding this split of responsibilities? Nov 24, 2020 at 14:26
  • Regarding observability I fully agree with what you said, we are lacking a shared view of logs. Nov 24, 2020 at 14:28
3

Should the bug from C be covered by proper unit tests and integration tests, such "cascade" issue could be mitigated.

Another thing is to setup proper dashboard for each Microservices, so non-200 response metrics could be configured to trigger pageduty alert. In your specific example, service B owner will be notified by the issue and start to investigate on it earlier.

Other than the above two, cross-team coordination and clearer responsibilities as other answers mentioned is also important.

3
  • Any hint about how others do cross-team coordination? The landscape right now is just composed by the teams, there is no organization outside these teams as following Agile practices we were supposing them to collaborate as peers to solve these situations. Nov 25, 2020 at 8:05
  • I was mainly referring to the operational process. Some organizations have technical product manager whose partial job is to bridge different teams with clear responsibilities on development and operation, some do not. Different teams just need to discuss on the procedure of handling a alert ticket and the escalation rules.
    – lennon310
    Nov 25, 2020 at 16:04
  • Application logs from different service are usually stored in internally accessible instances, but infra logs are usually accessible only by Techops teams. So there may be some process involved with them too (e.g., if there are proper metrics at platform level, when should oncall escalate the alert to Techops, etc)
    – lennon310
    Nov 25, 2020 at 16:06
0

It looks like you have a single product maintained by multiple teams.

In this case, you should check LESS (probably is preferable for your case) or Nexus approaches to scale Scrum which will help you to have layer-based team responsibilities but still have all teams working on the same product and properly coordinated.

This will enable you to have "Overall retrospective" - the formal event where all teams can discuss communication/dependency/tech/etc. issues and discuss solutions for them. As all teams have coordinated Sprint with a shared goal, this also helps them to have better coordination and communication for such things related to multiple teams.

As of the issue itself (finding root cause and responsible team) - there are various approaches you can apply:

  1. Important: Have session/request ID to be able to group logs of various layers to the same log thread (and then you will see which layer was the first one to report an issue, which can speed-up finding a root cause) for specific user request.
  2. Make sure to include all the logs in the same location so that all events of the log thread are available in one place.
  3. Make sure every layer has its own validation, sanity check, etc. so that you better understand if there is some system error with a microservice itself (and then this is, say, team A) or its a client which has sent bad request (and then this is, say, team B).

Hope this helps.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.