Why employers need WAA Certified Web Analysts

My personal experience of the WAA Certification test, by Vicky Brock

Earlier this week I was among the first brave bunch of people to take the WAA Certified Web Analyst test, when it ran for the first time at eMetrics San Jose.

And yes, for the record, that involved me getting my credit card out and investing hundreds of dollars of my own money.

And no, I don’t regret a single cent of it.

I am a web analyst in my job, but these days I am also an employer.  By taking the certification I was able to experience for myself what a balanced and challenging test this was and what being a Certified Web Analyst means from both a practitioner and employer perspective. 

Most significantly, it is not a test of book learning, but of analytical thinking and skill. 

My immediate impression was that I want to employ people that achieve this Certification because for the first time I have a differentiator between people who can use a tool and people who can actually use data to make decisions and recommendations. 

If you can pass this test, specifically the case study section, I know I can trust you to get beyond noise and data and KPIs and get to the heart of delivering meaningful business analysis to our clients.  I know that you should be taken seriously as a candidate for a mid level analytics position and above.

My impressions as an employer

As an employer, the fact that the certification is built to ISO standards and is legally robust in the way I can integrate it into formal procurement process is the icing on the cake.  This is particularly relevant as I work for government and corporate clients who are part of quality assurance schemes themselves.  The QA standards behind this test is a useful differentiator to me, whereas I can’t really do anything with a test that someone could have taken online, with their browser open, using info they crammed on a website. 

So there is value for me that this is a secure test, which I know the person in question took.  Passing this proves you can take real world data, real world business contexts, goals and even political challenges and interpret that correctly.

This is not the analytics equivalent of a swimming certificate.  I’d compare it with my experience of taking the GMAT, often used for MBA entry.  Nor is this test like the Google Analytics IQ test or similar. And this is not a certificate from a private consultancy firm or an industry celebrity saying you did great.  There are plenty of these kinds of awards and no doubt with every passing month the bandwagon will keep rolling and there’ll be plenty more. And those certificates are terrific in their context; they demonstrate interest, enthusiasm and commitment.  But in my opinion they are not a formal professional certification delivered to robust procurement standards.

What I like about this as an employer is that it is formal certification that the industry body representing the analytics field endorses that you have specifically achieved the challenging analytics criteria it has defined and I can review.  An independent body is verifying and keeping a record of the factor that the people it endorses have formally obtained and maintained their skills.  Organisations that have no analytics experience themselves can now employ analysts with some of the risk removed, because this certification follows the same standards and quality processes that certification of marketers, accountants and other competencies follow.

The experience from an analyst’s perspective

As a web analyst taking the test, I found it challenges not just your breadth but depth of knowledge.  I definitely recommend reading the  Knowledge Required For Certification document in advance. It will help you assess if you’re ready and if there are gaps to brush up on. (I skipped the conversion party to swot up on email, for example).  The first half of the test is a series of multiple choice questions spanning a really wide range of topics.  Some of these were not my field of expertise and I know I guessed at a few answers.  

More interesting is the second half, as it really tests your ability to take complex case studies with a great deal of both relevant and irrelevant information and to make analytical decisions based on that.  Context and goals has to be applied to reach the right answer, because all the multiple choice options associated with the case studies are valid under certain circumstances – but there is only one right answer that exactly meets the complex specifics given.

My end feeling on completing the test was very Rumsfeldian. I know there was plenty of stuff I knew.  I know there were a few things I didn’t know.  But what I will be really interested in is all those unknown, unknowns!  Talking to someone afterwards, they commented how they really couldn’t choose between B and C for one question.... hmm, and there was I wrestling between A and D. 

I look forward to getting my report breaking down the scores for the different sections and when I look from an employer perspective at candidates who’ve taken this test I’ll be particularly interested in seeing strong scores on the case study side.

This first test was paper based (whereas the roll out will be computer based at test centres) so I don’t know yet if I passed.  I hope I have, of course, but it’ll all come down to those unknown unknowns.  But even if I haven’t it has been extremely valuable to do as an employer of analysts.  Because if I haven’t passed myself, then I sure as heck want people who have!

The test will be running at eMetrics London next week (19^th May). If you’re thinking of taking the test or have any questions at all (beyond what are the answers, as I sure as heck don’t know those!) then please feel free to drop me a line or tweet @brockvicky

And as one employer to another I’d say this really is a differentiator - you really want the people that ace this Certification working for you!

Web Analytics Championship Judging and Winners

The WAA Championship is an annual activity where members are invited to review the association’s site and traffic data and put their analyst skills to work. After some hiccups with the access to data that delayed the starting date, the analysts analyzed, submitted their reports, insights and observations about the WAA site. The judges deliberated and had to take difficult decisions. Below we present the criteria used to judge the submitted papers and, of course, the winners.

Competition Criteria

There were four criteria against which the analyses were judged:

1. Business Goals: Were they clearly identified? (25 pts)
   Macro goal: Membership
   Micro goal: Member value, Recency, Frequency, Engagement
   Note: It was OK to not identify the macro and micro goals, but it was important to clearly identify some business goals.
2. Analysis: Was there enough data? (25 pts)
   Segmentation to provide context
   Correlations or other demonstration of relationships
   Multiple data sources used (more than just the web analytics tool)
3. Recommendations: Were the insights valuable? (25 pts)
   Meet SMART criteria (Specific, Measurable, Actionable, Realistic, and Time Bound)
   Informed by data
4. Overall quality (25 pts)
   Is the analysis structured, succinct, and focused?
   Appropriate use of visuals and data (graphs, charts, etc.)
   Writing and presentation quality

Fatal Flaws

After analyzing the entries, the judges agreed that some mistakes were so significant that if made hurt the credibility of the analysis, and in turn, would make it unlikely that the recommendations would be acted upon. Entries that contained a “fatal flaw” were eliminated from consideration. These “fatal flaws” were:

• Unfocused Analysis: When the focus of the analysis is not tied to business or marketing objective(s), the analysis is at best interesting trivia, but not important insight.
• No Compelling Story: The judges were unable to come to consensus on what it means to “use the data to tell a story.” However, they easily reached consensus about which entries had this fatal flaw. In some cases, the analyses with this fatal flaw were written stream of conscience and difficult to follow. In other cases, analyses with this fatal flaw were so boring that they failed to instill a sense of relevance, importance or urgency.
• Factual Errors: One bad apple spoils the bunch. One data error calls the rest of the data into question (or more importantly, the recommendations based on that data). It was particularly easy to spot factual errors when the analyses were mostly based on the same data sets.
• Unfounded Recommendations: The judges agreed that it is easy to report data and it is easy to make recommendations. However, limiting your recommendations to those based on the data is harder than it looks, especially when all the data that you want isn’t available. The result is unfounded recommendations that undermine a data-driven culture. Many of the unfounded recommendations in the analyses were usability related, such as recommendations to shorten the registration form. This would be a valuable recommendation if it were based on data that indicated that the length of the form was the key obstacle to completion. Or, even if there was data to indicate the form is a key obstacle in the member registration process. In fact, that data might exist, but it wasn’t referenced in the analysis.
• Inappropriate Tone: Entries were given much leeway, but the judges agreed that it was inappropriate for an analysis for this competition to be condescending or whimsical.
• Excessive Typographic Errors: Having one typographical error is not a fatal flaw, but an abundance of typographic errors demonstrates inattention to detail and makes it difficult to trust that the recommendations were based on good data. Blatant typos in the file name became a particular pet peeve, but didn’t eliminate anyone from consideration.

The Winners

Unfortunately, the majority of the entries didn’t make the cut because they contained one or more fatal flaws. Of those who met the criteria, there was a wide margin between the top two finalists and the rest of the entries. So, rather than select five finalists, the judges decided to only select two. Despite the focus on data, the winning entries were outstanding because of feelings they inspired. The winning entries were compelling and motivating. The judges immediately wanted to share the analyses with the WAA board members and discuss how to begin implementing the recommendations.

You can read the winning analyses here:

1. First Place: Submitted by Joel Wright and Ed Wu, both from the Dell Online Analytics Team
2. Second Place: Submitted by Leo Burnett team

Of course, if you’re interested in helping the WAA implement any of these recommandations, please leave a comment in this post or contact us by email.

The Judges & Team

The WAA is a volunteer-based organization and we want to thank the following:

• Judges: Jennifer Veesenmeyer, Judah Phillips, Stéphane Hamel and "Wandering" Dave Rhee.
• Logistic & Organization: Daniel Waisberg, June Dershewitz, Clare Madden, Shannon Taylor, Suzanne Carter-Jackson.

Message from the Chairman

As an independent, non-profit organization, the WAA is sometimes asked for its position on relevant topics in the web analytics industry. Privacy is important to all of us as a misunderstanding of our tracking technologies can lead to distrust. Therefore, I wanted to share a recent question about page tagging and web beacons being mistaken for spyware as well as the WAA's official response.

Haiku Wordle Twemes and Twitter Tweets

Saying this sentence:
"I wordeled emetrics tweets"
Sounds like Dr. Seuss

I Love Charts

Just came across a masterpiece on children education for Web Analytics :-) This short video is great: you can learn the song by heart and sing it in work meetings in order to convince them about the importance of graphs! Let me know how it goes...

New Standards Definition Doc

Over the last year, the Standards Committee have been hard at work. We took the definitions document from last year, and have totally revamped the definitions based on your feedback. We've added a few new definitions, but one of the things we're most excited about is our new "Ask Your Vendor" questions, where we've highlighted places where different vendors do calculations in different ways.

Results of Q1 2008 Web Analytics Communication Survey

Back in February, I launched a brief survey too see how web analytics practitioners are communicating our data within our organizations. A mere three months later (sorry! I've been busy!), I'm happy to report back the results. The four questions in the survey as follows:

Don't Lose Sight of the Big Picture

I recently completed an enormous effort in analytics. We launched a series of tests some weeks earlier and it was time to make an account for it all. One of the happy agonies of using a robust analytics tool is that you can look at the data from hundreds of different angles, each shedding a slightly different light on the subject. Anyway--I just wanted to make a case for stepping back and overlaying the tests on the big picture, rather than focusing purely in the world of each individual test.

Real-time Information

As part of this ongoing series of posts on communicating analytics data, I launched a survey back in February to see how web analytics professionals are communicating with their executive teams. I'm going to step through the entire survey results in my session at eMetrics Marketing Optimization Summit in Toronto, Canada before unveiling on this blog, but it turns out that most of the respondents communicate site visitor analytics on a weekly or monthly basis. This seems like an entirely reasonable span of time, but does it work for all KPIs? What about your most important metrics? How many times have your executives come to you out of cycle for a daily update — even when they'll be getting an update at the end of the week?

Video Metrics are a Focus for Measuring User Generated Content

I have come across a couple of discussions about measuring User Generated Video Content recently, and thought I would share them with members.