Skip to main content
added 10 characters in body
Source Link
John Wu
  • 26.9k
  • 10
  • 68
  • 92

Technical answer

The exact concept that is needed here is fault tolerance. You need a pipeline that is resilient to failure. This is an open-ended engineering problem, of course.

The most obvious brute force solution is redundancy, i.e. have redundant nodes in the pipeline so that even if something fails there is another server that can take over.

Another important but often neglected aspect to consider is the recovery mode. If there is a failure, does the pipeline pick up where it left off, or does it always start again from the beginning? Are build tasks idempotent and capable of being run repeatedly without issue, or do you have to trigger a full rollback and clean the system every time? This can make a huge difference in any process that has continuous problems.

Business answer

The other team needs help in getting visibility to the issues that are preventing them from deliveing a high quality, stable pipeline. Help them by gathering metrics of downtime and estimating the actual cost to the business. This will help them make a case with their own management to get additional resource (staff, hardware, training, or third party services) to meet the business' needs. You can also help with measuring success by coordinating and agreeing on an internal SLA and tracking their ability to meet it.

Needless to say, do not be adversarial about it, as this is often counterproductive.

Technical answer

The exact concept that is needed here is fault tolerance. You need a pipeline that is resilient to failure. This is an open-ended engineering problem, of course.

The most obvious brute force solution is redundancy, i.e. have redundant nodes in the pipeline so that even if something fails there is another server that can take over.

Another important but often neglected aspect to consider is the recovery mode. If there is a failure, does the pipeline pick up where it left off, or does it always start again from the beginning? Are build tasks idempotent and capable of being run repeatedly without issue, or do you have to trigger a full rollback and clean the system every time? This can make a huge difference in any process that has continuous problems.

Business answer

The other team needs help in getting visibility to the issues that are preventing them from deliveing a high quality, stable pipeline. Help them by gathering metrics of downtime and estimating the actual cost to the business. This will help them make a case with their own management to get additional resource (staff, hardware, or third party services) to meet the business' needs. You can also help with measuring success by coordinating and agreeing on an internal SLA and tracking their ability to meet it.

Needless to say, do not be adversarial about it, as this is often counterproductive.

Technical answer

The exact concept that is needed here is fault tolerance. You need a pipeline that is resilient to failure. This is an open-ended engineering problem, of course.

The most obvious brute force solution is redundancy, i.e. have redundant nodes in the pipeline so that even if something fails there is another server that can take over.

Another important but often neglected aspect to consider is the recovery mode. If there is a failure, does the pipeline pick up where it left off, or does it always start again from the beginning? Are build tasks idempotent and capable of being run repeatedly without issue, or do you have to trigger a full rollback and clean the system every time? This can make a huge difference in any process that has continuous problems.

Business answer

The other team needs help in getting visibility to the issues that are preventing them from deliveing a high quality, stable pipeline. Help them by gathering metrics of downtime and estimating the actual cost to the business. This will help them make a case with their own management to get additional resource (staff, hardware, training, or third party services) to meet the business' needs. You can also help with measuring success by coordinating and agreeing on an internal SLA and tracking their ability to meet it.

Needless to say, do not be adversarial about it, as this is often counterproductive.

Source Link
John Wu
  • 26.9k
  • 10
  • 68
  • 92

Technical answer

The exact concept that is needed here is fault tolerance. You need a pipeline that is resilient to failure. This is an open-ended engineering problem, of course.

The most obvious brute force solution is redundancy, i.e. have redundant nodes in the pipeline so that even if something fails there is another server that can take over.

Another important but often neglected aspect to consider is the recovery mode. If there is a failure, does the pipeline pick up where it left off, or does it always start again from the beginning? Are build tasks idempotent and capable of being run repeatedly without issue, or do you have to trigger a full rollback and clean the system every time? This can make a huge difference in any process that has continuous problems.

Business answer

The other team needs help in getting visibility to the issues that are preventing them from deliveing a high quality, stable pipeline. Help them by gathering metrics of downtime and estimating the actual cost to the business. This will help them make a case with their own management to get additional resource (staff, hardware, or third party services) to meet the business' needs. You can also help with measuring success by coordinating and agreeing on an internal SLA and tracking their ability to meet it.

Needless to say, do not be adversarial about it, as this is often counterproductive.