Traffick - The Business of Search Engines & Web Portals
Blog Categories (aka Tags) Archive of Traffick Articles Our Internet Marketing Consulting Services Contact the Traffickers Traffick RSS Feed

Archive for August, 2013

Testing and Statistical Confidence: The Three Bears

Monday, August 26th, 2013

The primary danger in taking actions in response to data (or even structured tests) in our industry appears to be overconfidence in overt data points and a lack of grasp of randomness and statistical confidence levels, leading to a flurry of actions and tweaks that get us more lost in the woods, not less. (I pointed this out here, last week.)

According to cognitive scientists like Daniel Kahneman and Amos Tversky, humans — even professional statisticians — are “poor intuitive statisticians.” (Predictably, some up-and-coming scholars in the same field have argued just the opposite.)

Translated into terms we see in our daily work in performance-based ad testing (etc.), it’s very common for novices — or even good workers who feel under the gun from bosses/clients who push us to “test more, do more,” — to respond frequently to random bits of data. For example, ads will be paused in favor of the ad or two that are “winning,” despite the statistical confidence levels on the win being (if you took the trouble to run them through a calculator) below 70%, despite the fact that a suboptimal attribution model is used such that sometimes segments get “last click credit” for “converting,” but other times do not (and again, often that is for no rhyme nor reason other than pure randomness). “What’s working better” in a combination of ads, keywords, queries, landing pages, inter-family buying dynamics, and medium to long consideration cycles — is never as easy as it looks. Tweaking to random data is, arguably, tantamount to concluding tests before they’re finished. In other words, you set up a whole bunch of experiments, and then shut them down prematurely. Wasted resources and insufficient learning/takeaways.

Too much random tweaking, we might say, is the action of the Impetuous Bear.

At the other end of the spectrum is the Inertia Bear. That bear once saw an A/B test that was rigged to be exactly the same ad competing with itself. 9 conversions accrued to the “winning version,” and only one to the “losing” (yet identical) version. Consulting the math experts, that outcome (in the case of a truly fair coin flip) happens only ten times out of 1024, so it’s less than 1% likely to happen. And yet it happened! From this, the Inertia Bear decides to insert a lot of these “placebo tests” into testing as a way to guard against acting on purely random results. Over time, though, the paranoia about some results necessary being random or impossible to explain (or correlate with the triggers being tested) begins to creep into a general mistrust of testing. That leads to a broader trend away from building anything new. That works fine, until it doesn’t.

So is the answer to simply be “Moderate Bear” and chart a path in between the two? Well, certainly you want to avoid either of these two extremes.

But in addition to that, you probably should be Curious Bear or Creative Bear, the kind of bear that fashions new things to see what might come of them, regardless of what the data say. If you’re purely driven by spreadsheets, provable outcomes, and “what’s best for the shareholders,” your output is bound to be less interesting. (See If Steve Ballmer Ran Apple.)¬†And the end result of that, we see all around us. It’s why — despite not being an Apple guy either — while I have been somewhat intrigued by the Microsoft Surface tablet, I never bothered to actually buy one. If someone didn’t demonstrably pour some passion into the conception and development of the product, then why would I line up to buy it? Probably, I eventually will. Maybe. (Is that level consumer intent even worth testing around?)

The Relentlessly-Widening Performance Gap

Wednesday, August 14th, 2013

Recently, I had a chance to review a deck produced by one agency handing back over a PPC account to its client owners (or potentially, a future agency).

Bullet point after bullet point referred to emphatic actions taken in the account. No doubt this only scratched the surface of the great number of actions taken.

I’ve always worked in the AdWords auction with a tacit ideal of “the perfect account.” Of course, there is no such thing. But insofar as every action has a chance to move either towards or away from perfection (much like a bronze sculpture, except that PPC is 90% science, 10% art), the end result of a vast many changes can vary greatly from manager to manager, account to account.

Analogy #43,589: you can think you’re going in a straight line in the forest, when what you’re really doing is making so many wrong turns you get impossibly lost. Getting out of the forest is actually easy, though, if all goes reasonably well. You find a river and follow it. Not counting the potential hypothermia and bears, “lost in the forest” is a picnic compared with “made 5,000 wrong moves in AdWords.”

What if the majority of changes in the account are pure guesswork, such that no change — the status quo — would have been superior to the change? As performance challenges toughen, are those errors compounded by further changes and desperation borne of declining performance? That sense of urgency — ironically, created by an urgent sense that bold, decisive action needs to be taken (often in the absence of insight into how much of that action is based on inexperience or guesswork) — parallels the “Death Spiral” concept that Jim Collins outlines in books like Good to Great. (That’s the opposite of focusing on the Hedgehog Concept and Turning the Flywheel.)

When you see a list of initiatives being taken in account management, it’s worth asking: how much of it is pure guesswork? Guesswork is nothing like testing. A testing orientation will greatly reduce guesswork. A typical “guesswork” bullet point would say something like “paused all ads with CTR below 1%.” Testing isn’t like that. Try instead: “Chose the winning ad, based on a thorough understanding of performance criteria, not necessarily reduced to one variable in all cases, and insofar as feasible, with statistically significant data (to a 95% or 99% confidence level).”

Other guesswork gimmicks include adding 1,000 new exact match keywords to an account from a keyword dump, because you have a theory about exact match. Despite 30% of those phrases being poor fits in the account. One step forward… two steps back… endless busywork… not data-driven.

When accounts are managed based on pure guesswork, they get farther and farther from perfect, rather than the other way around. Approaching perfection isn’t easy. Even the best account managers using the smartest rules will make many so-so decisions. But they need to be vastly outweighed by a strong overall sense of how moving parts in an account strategy fit together. And most of all, the actions taken should be accurate (not guesswork) most of the time.

Not as easy as it looks. It’s especially daunting to decide (many times over) that “no action” is safer than “wrong action.” How can you improve anything unless you’re constantly doing stuff, right? (Unless it’s the wrong stuff!)

If you hand a high quality account over to a relative novice, it’s predictable what will happen: performance will begin gradually deteriorating from Day One. As things worsen, the scattershot “panic tweaking” begins. It then takes a steady hand and a high quality rebuild to bring everything back into line.

Give an account to a guesswork-driven manager to build from the start, and there’s a good chance it won’t reach anywhere close to its potential.

As one of my colleagues said to a client, once: “It’s like chess.” (Unfortunately, turned out the client was a Chess Grandmaster, and far smarter than 99.99% of the population at chess. But that’s a story for another day.)

Manufacturing Purple Cows When Others Can’t

Monday, August 12th, 2013

“All they care about is price.” Seth Godin reminds us that pure commodity thinking is deadly. If that’s all your customers can think about, they don’t care about anything to do with your product.

As PPC marketers wading into competitive (sometimes expensive) keyword auctions, we’re dead before we begin if we can’t suss out some so-called Unique Selling Propositions. Call it an identity. Call it differentiation. Call it a “story” (All Marketers Are Liars). Or just call it a Purple Cow.

Looking back over the long haul of efforts to win through sheer tactics, I can’t find too many businesses that have been unbendingly commodified to the point where we just throw up our hands, helpless. There have been a few. Web hosting is so bad it will make your eyes bleed if you’re a PPC marketer. Obviously, stuff like credit card deals and comparison sites will get pretty ridiculous as well. And even if you’re really good at working hard on the details — endlessly tweaking landing pages for conversion, even — you’re up against other companies that have Tim Ash or Conversion Rate Experts working on the same thing, up against obsessed PPC optimization and testing from the likes of yours truly (Page Zero Media), etc. It’s no picnic.

We were pretty impressed when one of our former clients, 50% of whose revenues came from various takes on web hosting, threw up their hands and divested all of their web hosting businesses in order focus on areas that were working, areas that weren’t so ludicrously commoditized.

Even at that, their other line of business — domain names (in various guises and areas of that industry) — is supposedly dangerously commoditized due to the well-known “GoDaddy effect.” The GoDaddy Effect has reached way beyond GoDaddy of course. Behemoths like Yahoo, various utility companies, web developers and yes hosting companies by the thousands — are all in on the act.

We certainly found that turning a profit on an $8 or $15 domain name registration wasn’t going to be a walk in the park via PPC. Over the years, we had our successes and failures in that endeavor. But the client, fortunately, kept building their unique take on their services, and an extremely loyal following. Plain and simple, what they offered was better and more tailored to the discerning customer’s needs. The sometimes-higher price point was simply a signal of that better neighborhood. And indeed, nearly 100% of the customers are buying from our client as a conscious protest against the GoDaddy’s of the world.

After years of staying the course, building great products, and telling their story in a supposed “commodity” business, this client has experienced a nice little breakout. (Their stock price is, in fact, up 82% in the past year, and business has never been better.)

Drawing out your unique qualities is almost always vitally important unless your competitors (and their agencies or in house professionals) are just stumblebums when it comes to logistics and campaign optimization. And it is almost always possible.

Like Seth says, it’s pretty rare that you can purely out-execute your industry peers by managing strictly to numbers. That is tablestakes, to be sure. But your big breakthroughs come from figuring out how people are going to react emotionally to your offering. Yes, emotionally.


Traffick - The Business of Search Engines & Web Portals


Home | Categories | Archive | About Us | Internet Marketing Consulting | Contact Us
© 1999 - 2013, All Rights Reserved