October 2nd 2016

Lessons learnt from my first year using Optimizely

Written by Ian Causton

In the words of Nigel Tufnel, this goes up to 11!

I've been working with Optimizely for over a year now and have learnt a great deal. I wanted to share these learnings in the hope of helping others embarking on a similar journey:

  1. Leverage the power of Project Settings

    There is a fair amount of code re-use within any project, whether it be solving flicker issues or pushing custom goals. Storing these within the project settings allows them to be called via the global window object, so with a common function such as this:

    window.optlyReadCookie=function(name) {
      var nameEQ = name + "=";
      var ca = document.cookie.split(';');
      for (var i = 0; i < ca.length; i++) {
        var c = ca[i];
        while (c.charAt(0) == ' ') c = c.substring(1, c.length);
        if (c.indexOf(nameEQ) === 0) return c.substring(nameEQ.length, c.length);
      return null;

    The function can be called like so - window.optlyReadCookie('cookieName')

  2. Use the chrome extension

    Although provided by a third party this a must when testing and verifying experiments. A common scenario I have found this really useful for is in verifying whether a variation is causing issues on production. By training the QA/PO???s etc how to switch to the original of any experiment the root cause, usually not a variation, can be ascertained and the right resources allocated. The extension can be found here.

  3. Use the QA Cookie?? ?? ?? ?? ??

    Initially, I began coding experiments on environments alongside the dev effort. However, at a certain point in time while I was tweaking things this created bugs that affected everyone. Therefore, in combination with the extension aforementioned I now always use a QA cookie when running on any environment. I only switch it off when everything has been signed off in production. This ensures everyone else is not affected by my work and provides an extra level of security on production.

  4. Optimizely support are worth contacting

    Sadly this is not always the case but submitting a support ticket is a good way to find the answers to those difficult questions. In my experience, they are quick to respond and always helpful.

  5. Archive draft and paused experiments on live

    Continually improving performance is a key driver for any site, as an Optimizely developer, one of my key goals has been to keep the snippet as small as possible. By default, Optimizely retains the paused and draft experiments in the snippet but this can be changed. For more information please see this post.

  6. Understand the reports

    As a developer it is important to understand the Optimizely reports, in my experience, this is the area you will answer the most questions on.

  7. Don't create custom fixes to flickering, use this!

    If you are using technologies such as Ajax or you are loading a lot of js libraries in the footer you are going to have many flicker issues with your experiments. I created a number of custom fixes until I found this.

  8. Learn the power of the API

    A great deal of the CRO effort is getting buy-in from senior management. To this end, it is important that they are kept informed of all the successes. The rich Optimizely API has allowed me to create reports/status updates on platforms such as Confluence to ensure that this wider group are kept informed of what is happening.

  9. Add a custom class to the html element

    I frequently need to apply custom styling to a variation and I have found the simplest way to avoid conflicts is to add an extra class like so.

       /* _optimizely_evaluate=force */
       document.documentElement.className    += ' optimizelyXXXXLoading';
       /* _optimizely_evaluate=safe */

    I use the experiment id in order to keep it unique.

  10. With good training 80% of tests can be created within the editor

    Optimizely is a great tool and, in my opinion, it should be used to empower non-technical members of the team e.g. marketeers, to create the experiments they want. With a clear process, training and technical support I believe this is achievable. Within an Agile team this has clear benefits and something I always strive to achieve.

  11. Get Certified

    In my experience, a lot of product certifications are simply a box ticking exercise that signify little genuine knowledge. This is not the case with Optimizely, you do have to know the product intimately in order to pass and it was a great path for me to get up to speed. Highly recommended.