8

I would like to know what strategies do you use to deal with A/B tests of your app and gitflow.

Overview:

We are a team of 6 programmers who develop and maintain a large App. So far we have worked on gitflow with a few add-ons on the process added by us which worked perfectly well for a couple years. In a simplified way, we use:

  • master branch (only code of published versions)
  • release branch which merges in master after final redundancy tests
  • hotfix which only interacts with master branch in extreme cases
  • develop which accumulates the developed modules as they are finished and tested, eventually merging into release.
  • /feature which is a group of features that branch from develop and once they are finished and pass the different stages of testing merge back into develop adding functionality
  • /fix_develop which is a group of features that contain fixes to bugs encountered from earlier versions that are not too urgent to start a hotfix.

Now, as the app evolves, with the UX team and other stakeholder teams we are adopting a stronger A/B testing strategy where new releases will have 2 versions, and based on how our users like one version or the other will become the final master version for as long as they make sense for our users.

Having explained that, the question is: What strategies have you used or recommend to manage code of A/B testing versions in gitflow?

The options I've considered are somehow inconsistent, for example branching A and B branches from master and then joining the release branch to one or the other, where I don't know how to deal with separating the code contained from release branch to feature branches. Another option is creating release A,and B branches and develop A and B branches which sounds like too much branches and confusion for my team mates.

I hear your opinions, thanks!

Update: The App we develop is an Android App and we are implementing the A/B testing using PlayStore platform for A/B testing, which requires to create two APKs and upload one of them with a rollout %. Also to keep things simpler, and since changes may sometimes be greater than just a button position, we decided not to add our own switch for A and B testing in one single APK.

  • Aren't these simply feature branches? – Robert Harvey Nov 30 '17 at 16:07
  • 1
    thing is that these branches will go to the store, so would have to break all the testing protocols if I just upload code from feature branches. what do you think? – alexm Nov 30 '17 at 16:09
  • You're going to go through the entire vetting process for a branch that may never become an actual product? Doesn't that essentially double your work? – Robert Harvey Nov 30 '17 at 16:10
  • anyways this is just the way I see it now, my idea with the post is to learn how other people deal with this matter and apply what I think would make sense for us – alexm Nov 30 '17 at 16:11
  • it does double the work, but there are essential things we need to test and prove if they work for our users that are quite complex, this is all done to enlarge the bottom of our active user's fanel, so it is a lot of work, but makes sense to test – alexm Nov 30 '17 at 16:26
11

The way I've always seen it done is to have a single code base capable of serving both pages/views/forms.

ie. Its Feature flagged and deployed with two or more configs, or the 'does the user get A or B' method itself is part of the app.

In this case you only have the one version of the code, so your source control doesn't come into play.

This is far more preferable than the alternative of keeping multiple versions of the codebase with different features. These tend to diverge very quickly and you end up having to implement the same features multiple times.

In general I would be careful and restrict the scope of differences A and B can have both so that they can be plugged into a generic switch method (view A or view B for example) and so that you can understand the reasons for any different stats you get out of your testing. eg was the increase in sales linked to the button colour or the pricing formulae?

Edit. for App clarification

You can still have features flags in a play store app and package the codebase into more than one apk.

  • 3
    Yes, this is what I was going to suggest: feature flagging. The only problem with it is it increases the complexity of your software, unless you intend to remove the (unused) feature(s) in the follow-on release. – Robert Harvey Nov 30 '17 at 16:12
  • 1
    yes, I'm not in favour of feature flags usually, but this is the way I've done it and seen it done. I'm sure we've all seen the horrors you can get into having multiple live versions of the same product in branches. So I would naturally shy away from that approach myself. – Ewan Nov 30 '17 at 16:16
  • 2
    you can still have config settings and get two apks from the same code base. Sure your app will be bigger, but.... maintaining multiple codebases is challenging to say the least – Ewan Nov 30 '17 at 16:22
  • 2
    surely that kind of flow can be abstracted into a view – Ewan Nov 30 '17 at 17:28
  • 1
    Yes, for an enterprise development team, I would suggest so. For me, in an intensively committing enterprise development environment, trunk base development flow is much easier. Check out here: trunkbaseddevelopment.com. Sam Newman had a nice talk about feature toggle in youtube: youtube.com/watch?v=lqRQYEHAtpk. – ivenxu Nov 30 '17 at 22:27
1

I agree with @Ewan that feature flags are the way to go. That way you can add something in the build process that will deploy APK for either A test or B test. But if want an idea that does not use feature flags, here is one that I have not tried.

You could pull the common code out into a separate library in a separate repository. Then, each version of the test has its own repository with its own gitflow and master branch that can be deployed.

This will take more effort in the beginning in that all the common code will have to be pulled out into library(s) then referenced in the repository. But after the initial setup, I believe this can streamline the development of new features into the A/B test.

**Again, this is only an idea and not something that I have done personally.

  • Yes that makes sence, we have a library with the main functionality (C++) and then consumer Apps (Android and iOS) that only serve as interface and service deployers, what we are changing is just the way the app is presented to the user, so yes, that would work for A/B tests in Android. However, I'm looking to have the code centralized in one repository, so I'm looking for a gitflow strategy in the middle of: "copying all branches for A and B" and "having just one A and B feature or normal branch which would mean a dirty repo". – alexm Nov 30 '17 at 17:38
0

Since your app is already running and has some users, I suggest one of the following options:

  • I strongly suggest that you do a design thinking before implementing a new big feature;
  • If you still need to do the A/B testing, I suggest you keep with normal gitflow and have an A branch and a B branch;

Source control

Considering A/B testing, what I want to make it clear is that:

  • A branch: contains code regarding A version of the app. EG: A_HomeActivity, A_SettingsActivity, etc.
  • B branch: contains code regarding B version of the app. EG: B_HomeActivity, B_SettingsActivity, etc.

From this point, you could use 2 strategies:

  • Create two totally different versions: A.apk and B.apk; or
  • Make the B branch to also contain code from A branch, and provide an app capable of using the two flows. The user downloads only one app and has the option to turn the new flow on for evaluation purposes. I've seen something like this done by Bitbucket, on which a totally new navigation was available for few users for evaluation; after some period this new navigation became the default one.

Either way, after the experiment you simply remove code for the obsolete feature/flow.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.