Expectation vs. Testing Reality


Posted by, Dash Stewart, Conversion Optimization Analyst What did Batman say to Robin before they hopped into the Batmobile? . . . . . . . . . Robin, get in the car. I sometimes tell jokes at stand-up comedy open mics. A side effect of this is that I now know a lot of bad jokes. The above is a joke I heard recently that the comic thought was really funny, but didn’t go over quite so well in practice. Telling a joke that you think will be really funny, and getting a lukewarm response can be a very awkward and jarring experience. The larger the audience and the quieter the response, the more embarrassing this whole experience can be. All this embarrassment and awkwardness comes from our expectations not matching reality. So why am I writing about telling bad jokes in a blog entry that is supposed to be about Conversion Optimization? shutterstock_117516274 The Conversion Optimization Team is constantly running multiple experiments to address the issue of expectation vs. reality. Recently, we ran a test between three very different pages (let’s call them Pages A, B, and C) all attempting to garner free trial sign-ups for an entertainment company. These pages implemented what many CO practitioners would call “best practices” in order to get a user to test out a free trial.  While we ran the test in paid search and gathered nearly 100,000 collective impressions of our pages, we also printed screenshots and asked Performics employees to guess which page they thought would be most successful at converting. Our office test was by no means strict and scientific, but the results were no less surprising. Expected: With a total of 105 votes, a huge majority of those polled guessed that Page C would perform the best with 66 votes. Perhaps this can be attributed to the flashiness of the page or the large centered Call to Action and Free Trial button. With only 10 votes, Page B’s Call to Action button is minimized and placed in the corner.  The page has much more text that its competitors. Page A has several large images and a scrolling carousel displaying new releases. It split the difference with 29 votes. Reality: We came upon our final results with over 95,000 visitors between the three pages, and over 13,000 free trial sign-ups collected in a two week period. The winner by a large margin in on-page conversions was the least expected in our poll, Page B. It garnered a 9% lift with 99% significance over both of its competitors (14.85% vs. 13.68% and 13.63% respectively). The page everyone thought would convert poorly turned out to be much more successful than its competitors. This emphasizes that best practices and expectations are usually only a vague guideline for what will perform well. Even the CO team could not predict the winner correctly prior to initiating the test; and, we have been tweaking and analyzing these pages for months. Testing has a specific value that rises well above and beyond an expert’s intuition or even a highly specific rubric of best practices. A key skill for marketers, (Conversion Optimizers especially) is meeting user expectation by providing a smooth and reliable experience. Part of being great at optimizing for conversions is learning from testing reality and managing your expectations, whether a page performs like you think it will or not. So, what can this CO underdog story teach us about stand-up? Let’s try one of those jokes we don’t think are funny. Who knows? Maybe it’ll kill.


Comments are closed.

Performics Newsletter

[raw]



[/raw]