Advanced A/B Experiments
GWO provides for an A/B style of testing "out of the box". However, sometimes you may find it does not quite suit your needs, or, you may need more control over the URL to which visitors get redirected. This article describes how to perform an A/B test where you have much more control over how the redirection to the alternative pages takes place.
The following is a technique for performing an A/B test such that you have an opportunity to dynamically participate in the construction of the alternative URL's.
First, instead of creating an A/B experiment, create a Multi-Variate experiment. Place the control script at the top of your test (A) page. Place the test page tracking script at the bottom of your test page and alternate pages (B, C, etc). Place the goal tracking script at the bottom of your goal page.
Now, instead of introducing the multi-variate sections scripts, place the following somewhere in your test page (I recommend it go near the control script):
<!-- utmx section name="page-url" -->
This HTML comment is very much like a section script in that it declares a section named "page-url", but does not modify the page or the user's experience at all.
Now, you can validate the test and goal pages and move on to specifying the alternate URLs which you want to test. Enter these into the GWO UI as the content of variations for the page-url section. Here I've specified a simple relative URL, but you can specify a complete URL if you want:
Before launching or previewing your experiment, place the following script immediately after the control script on your test (A) page:
At this point, you can preview and launch the experiment. It will behave just like any other A/B experiment.<script>
function filter(v) {
var b = utmx('variation_content', 'page-url');
var u = v[0].contents;
if (b && u.substr(0,7) == 'http://' && b.substr(0, 7) != 'http://') {
u = u.substr(7);
}
return u;
}
utmx('url', 'page-url', 0, filter);
</script>
You can see an example of this in action here as follows.
Test (A) Page (inactive)Note that that URL will take you to the A page without enrolling you in the experiment. In order to experience a possible redirect, remove the content after the # in the link's URL:
Test (A) Page (active)
The Filter Function
This script above will perform the redirection to alternative pages, should GWO decide that a given visitor is not to see the A page. Let's look at a bit more closely.
<script>
function filter(v) {
...
}
utmx('url', 'page-url', 0, filter);
</script>
The utmx function is defined by the control script. It is the main entry point for a variety of GWO functionality available in test pages. In this case, the first argument, 'url' tells the function that it should treat this experiment as an A/B experiment and perform a redirect if necessary. The second argument, 'page-url', is the name of the section which defines the alternative URL's. The third argument is a positional indicator and should be set to zero in this case. Otherwise, I will not describe it here.
The fourth argument is a filter function which you define and is called just before redirection takes place. It takes, as an argument, an object containing the redirection URL computed by the utmx function and returns the actual URL to which the user will be redirected. It is your opportunity to get involved in the form of the URL the visitor is redirected.
Before calling the filter function, the utmx function does a number of things to the target URL. First, it merges all query parameters of the current URL (document.location) with the query parameters of the target URL. This allows an alternate page to have the same information the A page has. For example, you might encode product ID's as a query param:
http://www.mystore.com?product=slinky
You might enter:
http://www.mystore.com/b-page.htm
As the alternative B-page URL for your experiment. Because you are testing all your product pages, you can only specify the B-page URL, sans the product. GWO will redirect to:
http://www.mystore.com/b-page.htm?product=slinky
Which allows your B-page to know which product is being queried and present that product in the context of the B-page.
The utmx('url', ...) function also looks at the URL and adds http:// to it if it does not already have it. Many times, this is fine, if you don't specify the protocol. For example:
www.mystore.com/b-page.htm
But it can sometimes get in the way. For example, you might want to specify (as I do in my example above), a simple, relative URL for the alternate pages:
b-page.htm
The the code in the custom script above will strip this away as needed:
var b = utmx('variation_content', 'page-url');
var u = v[0].contents;
if (b && u.substr(0,7) == 'http://' && b.substr(0, 7) != 'http://') {
u = u.substr(7);
}
return u;
If the "raw" version of the alternative in the variable 'b' does not begin with http:// but utmx's version does, then it will be stripped away. Finally, the URL in the variable u is returned where GWO will perform a redirection to it.
URL Customization
The filter function allows you to inspect and modify the redirection URL at will. To demonstrate this further, consider my example above where the product ID is a query parameter:
http://www.mystore.com?product=slinky
But, what if my site encodes the product in the path of the URL? Like so:
http://www.mystore.com/slinky/a-page.html
And, I want to test an alternative page, like so?
http://www.mystore.com/slinky/b-page.html
You'd might enter the alternative URL in GWO as:
http://www.mystore.com/slinky/b-page.html
But, because your test page will be called for more than one product, like:
http://www.mystore.com/tofu/a-page.html
You can't enter that URL, otherwise all users will see only the tofu product, regardless of which product they may have clicked on. Or, you might enter:
http://www.mystore.com/b-page.html
But, no product is specified here, and your web server might produce an error page.
What you need to do in cases like this is write some custom JavaScript which builds the correct URL. So, building off the last example, consider the following example. Let's say the the following URLS are two of among many products:
http://www.mystore.com/slinky/a-page.htm
http://www.mystore.com/tofu/a-page.htm
http://www.mystore.com/tofu/a-page.htm
And you specify an alternative URL like so:
http://www.mystore.com/PRODUCT/b-page.htm
The idea is while computing the URL to which a redirection will take place, inspect the current URL (document.location.href) for the name of the product, and replace the word PRODUCT with the name of the current product in the a-page. Like so:
<script>
var b = utmx('variation_content', 'page-url');
function filter(v) {
var u = v[0].contents;
if (b && u.substr(0,7) == 'http://' && b.substr(0, 7) != 'http://') {
u = u.substr(7);
}
var l = document.location.href;
var prefix = 'mystore.com/';
var i = l.indexOf(prefix);
var j = l.indexOf('/', i + prefix.length);
u = u.replace('PRODUCT', l.substring(i + prefix.length, j));
return u;
}
utmx('url', 'page-url', 0, filter);
</script>
This is very much like the first example above, but instead of simply returning the URL, we get the current product name and use it to replace the place-holder token, "PRODUCT" which is present in all alternative URL's. This allows us to redirect to the proper alternative URL, while preserving the current product the visitor is interested in.
You can see this in action here:
http://www.gwotricks.com/abadvanced/slinky/b-page.htm
Happy redirecting!
Where does the Control Script belong?
I frequently see test pages in which the control script is not placed in a good location. In this article, I want to talk about the things to consider when placing the control script into your test pages.
Latency
The presence of the control script in your page will introduce latency into the total load time of the page. When the control script executes, it generates a request for a Google resource called siteopt.js. The latency is attributed to the time it takes for siteopt.js to load. To demonstrate this, with Firefox, you can load siteopt.js in the presence of the Firebug add-on that can measure the amount of time that it takes the page to fetch various resources. For me, inside the Google corporate network, it takes on average about 20 milliseconds to load siteopt.js:
When I do the same thing from my home, it takes a little more time, about 36 milliseconds (I use a microwave based ISP, which adds a little bit of latency to everything):
In order to minimize this latency, Google distributes the servers that respond to siteopt.js requests all over the globe. This way, visitors from Mongolia to your test page don't have to load siteopt.js from a faraway server in the United States, they will probably get siteopt.js from a server in Asia, or Northern Europe.
Redirection
If you are running an A/B experiment, the control script may cause a redirection to happen if Google decides that this particular visitor should see a page other than the A page. This means that all the processing that the browser is doing when the redirect takes place will be aborted when the new page is loaded.
Other Resources
Given these aspects of the control script, it is very important that the control script appear before any references to external resources. These include CSS, script, image, objects, and the like. The reason for this is that if the control script decides to perform a redirect, all the time and work involved in loading these resources will be wasted and, most likely, performed again in the target of the redirect. This leads to increasing the total latency that the visitor experiences.
Displayable Content
Because the control script loads alternative content used in the display of the page, it needs to appear before the points in the page that potentially use this alternative content. Additionally, it is very important that the control script appear before any content in the page that is displayed to the user.
The reason for this is, again, latency. If the control script were to appear after, say, the first paragraph of the page, the user would see that paragraph, experience a very brief latency, and then the rest of the page would display. However, if the control script were to appear before this paragraph, then the window remains blank during the small latency, and then the page would render as a whole. This is a better experience for the user.
Also, a browser may spend less time laying out the page because there is no interruption of the display of the page.
Document Type Declaration
Many pages have a document type declaration. It may look something like this:
Browsers will change the way they parse an HTML file based on this declaration. In order to determine the type of a page, browsers will "sniff" for this declaration at the very beginning of the page. If they find a well formed declaration, then the parser for that document type will be instantiated.<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN" "http://www.w3.org/TR/html4/strict.dtd">
It is very important that the control script appear after any document type declaration. The reason for this is that browsers will only look so far into an HTML document when sniffing for these declarations. The presence of the control script before the declaration may cause the browser to not find the declaration and to choose the wrong parser. This can have devastating effects on a page, potentially rendering it unusable.
Conclusion
So, in a well formed HTML document, the control script should be:
- After any document type declaration
- Before any other resources (CSS, scripts, etc)
- Before any displayable content (text, tables, etc)
Poor Man's GWO/Analytics Integration
A while back, ROI Revolution (a Google Website Optimizer Authorized Consultant) came up with a technique for importing GWO information into GA (Google Analytics). This way, you can see how different variations of a web page/site will affect the various numbers that GA measures.
The original technique relied on munging through the __utmx cookie that the control script sets. I improved upon this by describing a technique which uses some new functionality I incorporated into siteopt.js. Their most recent description of this technique is here.
I've been giving some thought to this technique and wanted to 1) Improve the technique a bit, and 2) Suggest an alternative which has some significant advantages.
First, an look at my script for the ROI technique. The script I suggested is (note, all these scripts assume that ga.js and the control script have already been included on the page):
<script type="text/javascript">
if (utmx('combination') != undefined) {
var l = document.location, s = l.search;
s = s + (s.length ? '&' : '?') + 'combination=' + utmx('combination');
var pageTracker = _gat._getTracker("UA-XXXXXX-Y");
pageTracker._trackPageview(l.pathname + s);
}
</script>
This suffers from two things. First, global variables are introduced into the namespace of the page and, second, there is no specification of which experiment is annotating the hits. So, consider the following, modified, script:
<script type="text/javascript">
(function(){try {
if (utmx('combination') != undefined) {
var l = document.location, s = l.search;
s = s + (s.length ? '&' : '?') + 'combination=' + utmx('combination');
s += '&experiment=MyExperiment';
var pageTracker = _gat._getTracker("UA-XXXXXX-Y");
pageTracker._trackPageview(l.pathname + s);
}
}catch(err){}})();
</script>
Here I've enclosed the script in an anonymous function and a block to catch exceptions (should ga.js fail to load), and also added a query param describing the experiment being tested to disambiguate one experiment from another.
With this technique, you can go into your content report in GA and see how many visitors assigned to the various experiment variations actually end up visiting pages on your site.
This technique requires you to modify the GA tracking script on each page of your site which you want to track against an experiment. If you've already instrumented your whole site with GA tags, it might be cumbersome to edit them all, especially when you'll need to modify them again when you end and experiment, or start a new one.
Experimenting on the entire Site
The answer I've come up with to deal with these issues is to integrate GWO experiment information a different way. Instead of modifying each page URL with the experiment information, you can set the GA user variable with experiment information (assuming you are not already using the variable for other purposes). Consider the following script:
<script type="text/javascript">
try {
var pageTracker = _gat._getTracker("UA-XXXXXX-Y");
pageTracker._setVar("MyExperiment-" + utmx('combination'));
} catch(err) {}
</script>
First note that this script does not call _trackPageView. Its job is not to track anything, but to set up the user variable to contain information about an experiment. This variable is stored in the __utmv cookie, and every subsequent call to _trackPageView in a tracking script will communicate the value of this cookie to Google servers and annotate the page view with the value in the cookie.
This script needs to placed in a particular place in your pages. First, it must only be located in GWO test pages which have already executed the control script. The script uses the function utmx, which is defined by siteopt.js, which is obtained through the control script. Next, this script needs to be executed before any calls to _trackPageView in any other GA tracking scripts. It also needs to be after the inclusion of ga.js.
I recommend that you place this script after the ga.js inclusion script and before the GA tracking script in the following manner:
<script type="text/javascript">
var gaJsHost = (("https:" == document.location.protocol) ? "https://ssl." : "http://www.");
document.write(unescape("%3Cscript src='" + gaJsHost + "google-analytics.com/ga.js' type='text/javascript'%3E%3C/script%3E"));
</script>
<script type="text/javascript">
try {
var pageTracker = _gat._getTracker("UA-XXXXXX-Y");
pageTracker._setVar("MyExperiment-" + utmx('combination'));
} catch(err) {}</script>
<script type="text/javascript">
try {
var pageTracker = _gat._getTracker("UA-XXXXXX-Y");
pageTracker._trackPageview();
} catch(err) {}</script>
This way, the variable gets set before the tracking script which will use the variable annotate this page visit. Furthermore any visits that this visitor makes in the future (for up to an absence of two years) will also be annotated.
You can see this in action here:
Test Page
Special Consideration for A/B Experiments
The technique described above works for multi-variate experiments. That is, experiments which do not use redirection to present different pages to visitors. If you were to use this technique on GWO A/B experiment, the code which sets the user defined variable would sometimes not get executed because the original page would redirect to an alternate page, skip the setting on the original page for one which does not have the Control Script, and can't use the "utmx" function.
Now, there are is a way to have the Control Script on A/B alternate pages. I'll save that technique for another article. However, there is a way to get the above technique to work A/B (redirection) experiments. The idea is to set the user defined variable before the redirection on the original page takes place. To do this, consider the following A/B Control Script:
Notice that this is really two scripts. The first script is that which, among other things, defines the "utmx" function. The second, using the "utmx" function, is the one responsible for performing the redirect to an alternative page, should it be required.<script>
function utmx_section(){}function utmx(){}
(function(){var k='3923492669',d=document,l=d.location,c=d.cookie;function f(n){
if(c){var i=c.indexOf(n+'=');if(i>-1){var j=c.indexOf(';',i);return c.substring(i+n.
length+1,j<0?c.length:j)}}}var x=f('__utmx'),xx=f('__utmxx'),h=l.hash;
d.write('<sc'+'ript src="'+
'http'+(l.protocol=='https:'?'s://ssl':'://www')+'.google-analytics.com'
+'/siteopt.js?v=1&utmxkey='+k+'∓utmx='+(x?x:'')+'&utmxx='+(xx?xx:'')+'&utmxtime='
+new Date().valueOf()+(h?'&utmxhash='+escape(h.substr(1)):'')+
'" type="text/javascript" charset="utf-8"></sc'+'ript>')})();
</script>
<script>utmx("url",'A/B');</script>
It is between these two scripts where you will need to include ga.js and set the user defined variable in order for this technique to work for A/B experiments. So in the A/B case, move that code from just before the tracking script to be between these scripts.
Experiment without GWO reports
When you normally set up a GWO experiment, you are asked to place test and goal page tracking scripts on your test page and goal page. This allows GWO to track when visitors hit your test and goal pages, and render a GWO report.
If you only plan to look at the GA reports for this experiment, you can forgo placing the GWO tracking scripts. Instead, you need only place the control script and user defined variable script in any pages which have experiment variations. In fact, you can place these scripts on pages which do not have any experiment variations. The effect of doing this is that visitors to these non-variation pages will have their experiment variations chosen for them before they encounter any test pages. This means that you will be able to track their activity before they encounter experiment variations.
This is useful in that, if you do this at every entry point to your site, all tracking events will be tagged with experiment information.
Report Segmentation
So, once you get all this set up, how do get experiment segmented reports? On each report, there is a Dimension dropdown:
This will show a report broken down by the values of the user defined variable. You can use this to get an idea of how individual variations of your site influence this GA stat.
Note that this is not like a GWO report. These are just raw numbers and are not analyzed for confidence with respect to which variation is statistically significant.
Subscribe to Posts [Atom]
