A few months ago, I helped a Product Manager run extensive tests on an offering he hoped would make the site better. Together we tweaked the offering and ramped up volume on the test, until finally we stamped it as a tried and true initiative.

There was some backend work that had to be put into place to make its process work with all of our other systems, and this was no small chore for our development team. So there were a lot of eyes on this project from the moment we pushed it live (last week).

The boost in conversion rate and online revenues hasn't been quite as big as our tests suggested--not so far, anyway. And suddenly I found myself fielding a volley of questions about what other changes had been pushed to our site at the same time.

Yes, there were other tested initiatives that we pushed live with this release. Plus new tests were launched.

"Why?" my PM wondered. "Why can't we turn everything else off so I can know for certain how much lift my initiative has created?"

This is a conversation that I've re-played in my head many times since the actual conversation took place. Here are some of my best answers:

• We test so we know what to do next. Once we know, measurement takes a back seat to our actual goal: winning.
• Life moves fast. We would be wise to go down the road of more aggressive testing, meaning more concurrent tests, not fewer.
• If we isolated our test and control segments well enough during the tests, we can be sure we're driving in the right direction. If we don't think there was enough isolation around our variables, we can pull back and re-test.
• Testing is a poor man's time machine. The only way to know for certain what difference anything had is to go back in time and change that one thing, keeping everything else the same. If that's not in your budget, you test, analyze, and step forward into the vast unknown.

None of them are as pithy or persuasive as I want them to be (you can imagine how poorly the actual conversation went). Can you help me think of any others? Or maybe I'm going the wrong way with this--should we have a more robust strategy of isolating and measuring the initiatives we release?

Comments

[Avinash Kaushik]

Craig I think you should, and can, have a robust strategy around isolation. In some of my presentations I mention the concept of "triggers". Quite simply it means that rather than splitting all the traffic that comes to a page you split just one, or specifically identified, segment and leave all the other traffic alone.

So if you are testing on the home page then you only "trigger" traffic from a search engine into a test, or just your email / banner ad traffic etc.

The nice thing is that in this case you know exactly who you are sending into the test and hence you can isolate the number of other variables (mudiness!) and more confidently pontificate on the outcomes. :)

You can answer your PM by saying "we don't have to turn the other things off, let's send a controlled group into the test, split them, measure results, win, get bonuses!".

Some tools don't have the robustness you need to trigger people into a test, but other tools allow you a lot of flexibility (by sources, actions on the site, repeat visits, content consumed etc).

The only down side is tests might take slightly longer to product results. The nice thing is you can run a lot more tests concurrently.

Hope this helps a little bit. Keep the faith!

-Avinash.