A/B Testing with Optimizely

Does this red subscribe button get more clicks than the yellow one?

Which product description lead to more sales?

Does my author image frighten away readers or encourage engagement?

These are examples of questions that publisher and editors ask themselves about their websites. A/B testing - testing two or more versions of a page - is a way to answer these questions.

A/B testing has the potential to increase your website's revenue by allowing you to select content or styles that produce better results when compared directly. However, it can be tricky and we have seen several solutions come and go over the years. At this time we are recommending Optimizely but be warned that this is not a "set it and forget it solution."

Along with the whys and hows below, you will also find some points to be cautious about.

Create an account at Optimizely.com

Go to https://www.optimizely.com/plans/#testing to create an account and follow the instructions to set up your AB testing.

There is a definite learning curve here but you don't need to know anything about HTML or JavaScript as long as you stick to the step-by-step instructions.

When you are done creating the account and before you create your first A/B test, Optimizely will provide you with a snippet of code to place on your website.

Contact your ePublishing Project Manager

Jump on Basecamp and post a note to your Project Manager. They will need the snippet of code that Opimizely created above. The code needs to be placed as high up in the <head> tag as possible to work properly. This is something that a developer will need to do.

Your Project Manager can perform the work as an approved Change Request. It is a billable task, however it can be performed quickly.

Create an A/B Test at Optimizely

The nice thing about using Optimzely code compared to many other A/B test solutions is that you will manage the test through the Optimizely tools and not on the Continuum side.

Optmizely literally rewrites portions of the pages you are testing on when they are presented to your end users.

Once you have created and published your new A/B test at Optimizely you should see the different versions appear on your website and you should see results in the Optimizely reports.


  • Optimizely tracks which visitors have seen which version and attempts to make the experience consistent. This means that when you are testing you may have to clear your cookies and browser cache multiple times (note that your office may have additions caches further upstream in your network that will need to clear on their own) before you see all versions of your test.
  • Depending on the amount of traffic that your site receives, Optimizely may need to run for days or weeks before you get an actionable result. See Warnings Below...

Watch Out!

Our clients' experience with Optimizely has taught about a few gotchas. You need to keep the following in mind then performing Optimizely tests on your ePublishing website.

  • NUMBER 1 RULE! DON'T IGNORE THIS OR THE WORLD WILL END! I hope that got your attention because this is the one thing that we see go wrong. Our website code is updated from time to time for bug fixes, improvements and general tweaks. When you create your A/B test, Optimizely caches (grabs and stores a copy of) the code from your website. If your Optmizely test is coded against something that changes on our side this could create problems on your page. We have seen text layout issues and buttons and forms suddenly stop working because the code that Optmizely saved from your website is now old and incompatible with that is there now. SOLUTION: Don't run your tests for longer than 2-3 weeks at at time. Also, monitor the pages you are testing  every day to make sure they still work. If you see any problems, immediately suspend the Optmizely A/B test you are performing then check again. It may take up to 15 minutes for the problem to go away due to caching. Ignoring this advice can potentially break your website.
  • Be sure to double check the changes that you made to the website through optimizely. Remember that Optimizely is rewriting portions of your live website. If the layout is wrong or the Optimizely code is changing more than what it is supposed to, there is nothing that ePublishing can do about it. If you are testing with Optmizely and see layout or problems with buttons not working check your optimizely setup before opening a ticket at ePublishing. If you do open a ticket on our help desk which testing with Optimizely be sure to mention it in the support ticket. This will help us avoid hours of head scratching on our end.
  • Inline with the the Number 1 Rule above, when you are done with a test at optmizely, shut it down. Leaving tests running unattended is the surest way to run into problems.