I recently completed an enormous effort in analytics. We launched a series of tests some weeks earlier and it was time to make an account for it all. One of the happy agonies of using a robust analytics tool is that you can look at the data from hundreds of different angles, each shedding a slightly different light on the subject. Anyway--I just wanted to make a case for stepping back and overlaying the tests on the big picture, rather than focusing purely in the world of each individual test.

I found myself doing the opposite. I started within each of the test's little spheres: splitting our KPIs into the Test vs. Control group. From there I started dicing it up--what about people's immediate reaction to the test? Their latent reaction? What about people who engaged with the content vs. those who just saw the pages, or navigated past them?

Using this method, I produced hundreds of reports, then distilled my research into forty or fifty slides, then again mashed those down to a couple of summary and recommendation slides for each of the six tests we had running.

I sat on the finished analysis for a few days before I would formally present them to the group. Then I remembered to look at the big picture. What were our overall trends in the business? When I looked at those graphs, I was able to say everything that I had learned in those hundreds of reports with two simple graphs: the overall sales lines and lead gen lines. And once I looked at them, I said out loud: "Well, duh!" What we were achieving with our test wasn't what we wanted, but it was suddenly obvious what effect it was having and why.

So my advice to myself: don't get lost in the details. Better to start with the big picture instead of including it as an after-thought.

Comments

[Bogdan Secara]

Hi Craig,

I guess that's a good advice for all of us - thanks for sharing this here. May be the post should have the title "Don't Lose Sight of the main KPIs"...