Strategy for dealing with A/B tests and Gitflow

Question

I would like to know what strategies do you use to deal with A/B tests of your app and gitflow.

Overview:

We are a team of 6 programmers who develop and maintain a large App. So far we have worked on gitflow with a few add-ons on the process added by us which worked perfectly well for a couple years. In a simplified way, we use:

• master branch (only code of published versions)
• release branch which merges in master after final redundancy tests
• hotfix which only interacts with master branch in extreme cases
• develop which accumulates the developed modules as they are finished and tested, eventually merging into release.
• /feature which is a group of features that branch from develop and once they are finished and pass the different stages of testing merge back into develop adding functionality
• /fix_develop which is a group of features that contain fixes to bugs encountered from earlier versions that are not too urgent to start a hotfix.

Now, as the app evolves, with the UX team and other stakeholder teams we are adopting a stronger A/B testing strategy where new releases will have 2 versions, and based on how our users like one version or the other will become the final master version for as long as they make sense for our users.

Having explained that, the question is: What strategies have you used or recommend to manage code of A/B testing versions in gitflow?

The options I've considered are somehow inconsistent, for example branching A and B branches from master and then joining the release branch to one or the other, where I don't know how to deal with separating the code contained from release branch to feature branches. Another option is creating release A,and B branches and develop A and B branches which sounds like too much branches and confusion for my team mates.

I hear your opinions, thanks!

Update: The App we develop is an Android App and we are implementing the A/B testing using PlayStore platform for A/B testing, which requires to create two APKs and upload one of them with a rollout %. Also to keep things simpler, and since changes may sometimes be greater than just a button position, we decided not to add our own switch for A and B testing in one single APK.

Answer 1

The way I've always seen it done is to have a single code base capable of serving both pages/views/forms.

ie. Its Feature flagged and deployed with two or more configs, or the 'does the user get A or B' method itself is part of the app.

In this case you only have the one version of the code, so your source control doesn't come into play.

This is far more preferable than the alternative of keeping multiple versions of the codebase with different features. These tend to diverge very quickly and you end up having to implement the same features multiple times.

In general I would be careful and restrict the scope of differences A and B can have both so that they can be plugged into a generic switch method (view A or view B for example) and so that you can understand the reasons for any different stats you get out of your testing. eg was the increase in sales linked to the button colour or the pricing formulae?

Edit. for App clarification

You can still have features flags in a play store app and package the codebase into more than one apk.

Answer 2

I agree with @Ewan that feature flags are the way to go. That way you can add something in the build process that will deploy APK for either A test or B test. But if want an idea that does not use feature flags, here is one that I have not tried.

You could pull the common code out into a separate library in a separate repository. Then, each version of the test has its own repository with its own gitflow and master branch that can be deployed.

This will take more effort in the beginning in that all the common code will have to be pulled out into library(s) then referenced in the repository. But after the initial setup, I believe this can streamline the development of new features into the A/B test.

**Again, this is only an idea and not something that I have done personally.

Answer 3

Since your app is already running and has some users, I suggest one of the following options:

• I strongly suggest that you do a design thinking before implementing a new big feature;
• If you still need to do the A/B testing, I suggest you keep with normal gitflow and have an A branch and a B branch;

Source control

Considering A/B testing, what I want to make it clear is that:

• A branch: contains code regarding A version of the app. EG: A_HomeActivity, A_SettingsActivity, etc.
• B branch: contains code regarding B version of the app. EG: B_HomeActivity, B_SettingsActivity, etc.

From this point, you could use 2 strategies:

• Create two totally different versions: A.apk and B.apk; or
• Make the B branch to also contain code from A branch, and provide an app capable of using the two flows. The user downloads only one app and has the option to turn the new flow on for evaluation purposes. I've seen something like this done by Bitbucket, on which a totally new navigation was available for few users for evaluation; after some period this new navigation became the default one.

Either way, after the experiment you simply remove code for the obsolete feature/flow.