1

my company currently develops software without ever having releases. This makes all our customers live on the bleeding edge, and they know that.

Now we want to do releases, and intend to regulary fork from the branch, let that branch be testing and after some weeks of testing that branch becomes the new stable.

But how to I effectively fix bugs in the branch now? I could switch to the branch, refresh stuff in Eclipse ( 10 Minutes gone ), now fix the bug in the testing branch, test and commit it. Then I manually create a patch, switch to trunk, submit the patch and continue "normal" development ( Another 20 minutes gone).

I am concerned that the fixed I will do to the testing branch are too time consuming.

Are there better ways to do it?

6
  • 2
    but if 30min is 'too time consuming' I think you are going to be in for a shock when you start doing tested releases
    – Ewan
    Jan 11 '17 at 14:35
  • @Ewan gitflow? TO uses subversion. I agree on git beeing a better choice as a VCS, but gitflow doesn't help, if TO doesn't change his VCS ;) Jan 11 '17 at 17:07
  • you can still do a hotfix branch etc
    – Ewan
    Jan 11 '17 at 17:10
  • 1
    You can merge changes from one branch into another. So if you've fixed a bug in one, you can merge it across to the other. That may be quicker than manually duplicating the bug-fix, and will keep traceability in the SVN log.
    – Simon B
    Jan 12 '17 at 16:33
6

OK, I'm not upto date on your stack, but in my experience once you start talking about doing releases and reliability you are looking at days if not weeks of QA and release management.

The 30min of branching and and 1h of programming to fix a bug is dwarfed by testing. even if you automate it.

10min - Create hotfix branch from correct version of software
30min - Repeat bug locally
1h    - write failing unit test
1h    - implement bug fix
10min - run all unit tests locally
30min - merge hot fix branch back in, build in CI environment, run unit test
24h   - run automated UI tests
30m   - deploy to test/qa envrioment/machines
1wk+  - manual QA/exploratory testing and sign off
1wk+  - schedule live deployment date, rollback plan etc
1day  - manual deployment to live blue, watch graphs, bug reports etc
1day  - wait to see if there are bugs
2h    - manual deployment to green

We can improve on this with automated deployment tools and testing, but the thing we can't reduce is the time it takes for humans to gain confidence the the new release is 'safe'

To get around this people take one of two approaches

1: bundle up changes into one big monthly/quarterly release

This means you only run though the final test once (assuming no bugs) rather than per fix. It also allows you to assess the software as a whole, which tends to be more forgiving. ie well we still have this bug here, but overall its improved.

Another advantage is that you only have to experience the pain once a month, all the meetings etc become more routine and there is an urgency for the business to release rather than rollback and wait for the next monthly slot.

2: Split your app into lots of small services or components and release each separately

This means that each individual change is smaller and has a defined risk. It's easier to answer the question "what could go wrong" which means that some small changes can be released with less or no checking.

Edit:

Is it a waste of time? well I guess it depends on how much you value reliability. Fundamentally any change you do could go wrong in an unforeseen way. If you don't test that EVERYTHING works before you release a change, then something might not work.

But testing isn't perfect, so even when you think you tested everything, you may have missed something. You can only reduce the risk and weigh what level you are willing to accept to achieve the benefit of the change.

6
  • This applies if your product is a single multifunctional monolith. Componentized / micro-services approach lets you have much smaller testing surface before promoting a build to blue. For a reasonably built monolithic product, you don't need a week of exploratory testing for a small change, because change is compartmentalized and won't affect unrelated parts by design. Thus, doing a lot of small changes with a few builds in the testing pipeline may be more productive than doing rare huge cumulative releases.
    – 9000
    Jan 12 '17 at 16:13
  • 2
    "2: Split your app into lots of small services". unfortunately microservices dont help that much in practice. you still have to test your overall solution
    – Ewan
    Jan 12 '17 at 16:15
  • For the overall solution you likely have integration tests anyway? When you have a small number of well-defined interfaces, and you build other parts to these interfaces, you drastically lower the overall complexity of the system, because dependencies are fewer. This applies to microservices, this applies to monoliths architected the same way. It's when your dependency graph is looks complicated you have problems that changing a component may affect other, seemingly unrelated components in interesting ways.
    – 9000
    Jan 12 '17 at 16:20
  • 1
    sure, it helps. but its easy to create a 'distributed monolith' of services and orchestration and at the end of the day the same proviso applies. If you didn't test it, it might break.
    – Ewan
    Jan 12 '17 at 16:23
  • 1
    dev: hey manager we have to do all this list to make sure our app is working. Manager: ohh i see... now search for a way to do all of this in the 30 minutes i gave you. - true story.
    – linuxunil
    Jan 12 '17 at 16:54
1

How expensive are your bugs for your customers? That is, do they badly need them fixed ASAP, or would rather prefer stability in a status quo?

How expensive is a mistake in your product? That is, is shipping a bug a nuisance or is fatal?

For a product with expensive bugs (e.g. accounting software or aircraft control systems), you'd have long release cycles, rare cumulative releases, and a large share of time spent on creative manual testing.

Since your users live on the bleeding edge currently, the above must not be your area.

For a product with inexpensive bugs, you'd have very fast release cycles, mostly automated testing, and potentially multiple releases a day.

The smaller a release is, the less creative testing it needs, because a change is small and its impact anywhere else in the system is usually zero. The smaller is the set of changes, the smaller is the chance of their unanticipated interplay.

Also, the smaller is the change in the new release, the easier it is to roll back to the previous known good release. (Being able to do so in a matter of minutes is always useful.)

So, "release early, release often".

Get a role of a release engineer; usually people take shifts doing releases. The (today's) release engineer determines when to cut the release, that is, what changes are included in it, makes sure the build of the release branch is made and tested automatically, then does a few manual tests of the changes that don't have automated tests (e.g. subtleties of UI interaction).

If a release passes the tests, the release is promoted to the "blue" ("beta", "canary") state; a bit of production load is diverted to it, or it's made available for download by customers that signed up to beta releases.

If a release fails a test, the release is postponed, and the problem is communicated to whoever is responsible. If a fix is trivial, it can be backported to the release branch, and the process repeats. If the fix is large / takes long, the release can be canceled, or an earlier state of the master branch can be cut as the release branch.

To my mind, with a setup like this, a release should not be a big deal, something requiring a lot of intellectual effort. It will take time to handle all this, though. Allocate this time; measure the time actually spent, and adjust. The payout is fewer bugs shipped (as opposed to bleeding edge), and faster bug fixes (as opposed to releasing once a year). Usually it's worth it.

1
  • Thanks for all these comments. Indeed are most of our bugs not very expensive, and users live with them, but also get them fixed within 1-3 days from when they report them. But some also want the more stable releases. Currently we do exactly what you propose, we release early and often, but also I feel the need to create releases. Easpecially since we start to target the international marker, and releases would be a nice way to have marked builds that are stable and the translators also have time to translate everything while the build is in blue. Not really the A to my Q, but still useful.
    – Daniel
    Jan 13 '17 at 19:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.