5

I develop in an environment where we tend to do a lot of A/B testing. I am curious to learn from others on how A/B testing code is managed in your code base?

Do you mix your A/B testing code with your control code (existing functionality). If you do so, how do you ensure the code base from becoming unmaintainable?

Is there a specific design pattern you found to be useful in "forking" out control vs test code?

My current place basically asks front end engineers to keep copy the "entire" front end folder for each experiment and make modifications to it.

Pros:

  1. Isolates front end control code vs test. If test code has bugs, control code is not affected. You can turn off A/B test without doing code release.
  2. Can do ambitious layout changes with this pattern.

Cons:

  1. Change management becomes more difficult. If you are running 5 concurrent A/B tests, and makes modifications to the control code, you will have to ensure all A/B tests get the changes made to control code.
  • Look up blue green deployment. Combine that with decent source control and the most complicated part of A B testing is tracking the customers experience with the experiments. – candied_orange Sep 12 at 23:11
5

Four things helped me a lot when deploying many features per release, and testing them in production:

  • Feature flags.
  • Whitelists.
  • Routing control.
  • Detailed logging.

A feature flag allowed us to enable a feature for controlled time, e.g. a specific part of a day, or in one region only. It allowed us to quickly "undo" a change that did not work out for customers.

A whitelist allowed us to only enable a feature for a specific, well-defined set of customers. It allowed us to have explicit "beta-testers" of a feature, and quickly expand or contract their list.

Routing control allowed us to e.g. direct 2% of traffic to a server with feature set A, 3% to a server with feature set B, and the rest to the latest stable.

Detailed logging allowed us to tell which log line belongs to which server, which release, etc. Every line contained enough information to easily group lines with particular features enabled / disabled, and analyze / compare (using Splunk).

Hope this helps.

  • This is a valueable answer, though I fail to see how it answers the literal question. Or is the recommendation for "feature flags" implicitly a recommendation against "keeping a copy of the entire front end folder"? If so, consider to make this a little bit more clearer in your answer. – Doc Brown Sep 13 at 5:55
  • 1
    @DocBrown: If I had to state that explicitly, I'd say that the question is asked at an incorrect level of abstraction. Where you put your code is less important that how you structure feature activation, from my limited experience. – 9000 Sep 13 at 16:24
4

Dependency injection and the decorator pattern are your best friends for doing A/B testing. My team has several experiments running at any given time and we maintain a single codebase for all of them.

The general idea is to define an interface for every piece of logic that you would like to A/B test, and create:

  • a production (control) implementation with the existing logic
  • one or more treatment implementations with the new logic you want to test
  • a switching layer (decorator) that composes the above implementations and chooses which one to use for an incoming request

Combined with dependency injection, it becomes trivial to add and remove experiments without touching the control implementation.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.