Misusing Marketing Research: There’s a Saying for That

How to Lie with Statistics by Darrell Huff (1954)
How to Lie with Statistics, Darrell Huff’s business classic, first published in 1954.

Personally, I have respect for marketing research as a discipline.  I think most business decisions are better when they’re backed by the power of marketing research.

Still, I recognize that research can also be used in misleading or otherwise improper ways.

Even worse, research results can be contorted to justify business decisions that have been predetermined.  All too often, “How can we produce results that justify our position?” is the impetus behind a research initiative.

It’s that “dirty little secret” of research that was brought to light decades ago in Darrell Huff’s business classic, How to Lie with Statistics.  First published in 1954, this book been published in countless editions and remains in print even today, 60 years later.

Quirk's Marketing Research ReviewRecently, Dan Quirk of Quirk’s Marketing Research Review, the American research industry’s leading practicum publication, asked subscribers to share their favorite research-related quotes — ones that point to the folly that can be part of the discipline at times.

Some of the reader contributions are great — and they certainly point to the downsides of the research field.  Consider these bon mots:

“Science is built of facts the way a house is built of bricks … but an accumulation of facts is no more science than a pile of bricks is a house.”  (attributed to Henri Poincaré)

“Don’t let the facts get in the way of the truth.”

“When research walks on the field, judgment does not walk off.”  (attributed to Richard Kampe)

“Don’t theorize before one has data:  One begins to twist facts to suit theories, instead of theories to suit facts.”  (attributed to Sir Arthur Conan Doyle)

“Precise forecasts masquerade as accurate ones.”  (attributed to Nate Silver)

“If you torture a data set long enough … it will confess.”

“There are three kinds of lies: lies, damned lies, and statistics.”  (attributed to Mark Twain)

“Statistics can be misleading; the average human has one breast and one testicle.”

“A little nonsense now and then is relished by the wisest men. (attributed to Roald Dahl)

And this one, which ties everything up in a neat little bow:  “No research is better than bad research.”

If you have other memorable research quotes to add to the list, please share them with other readers here.  It’ll be good for a chuckle at least!

Mere Words? Google’s Library Project Speaks Volumes

Google Library Project
Google's Online Library Project: 5 million+ volumes and growing.
An article published recently in Science magazine provides fascinating sociological findings based on researching the content of the growing number of books in Google’s digital library.

Google has amassed a database of some 2 billion words and phrases from more than 5 million books published over the past 200 years. Much of the news coverage about this project has been focused on the intense criticism of some publishers and authors who are concerned about copyright protections and Google’s alleged knowledge “power grab.”

But a more interesting and useful result of Google’s library project has been that linguists have been able to use this trove to measure information and trends based on the language in the books and the people and concepts that are referenced therein.

By analyzing the digitized text of the books in Google’s database in relation to when they were published, the researchers found that they can measure all sorts of trends – such as changing tastes in foods, ebbs and flows in relations between countries, and the role of religion in the world.

For example, references to “sausage” peaked in the 1940s and have dropped off dramatically since then, whereas references to “sushi” began to appear in significant volume in the 1980s.

It’s also interesting to see how references to certain “personalities” grow or decline over the decades. Revolutionary leader Che Guevara was covered widely in the 1960s but has receded since then, whereas Hollywood actress Marilyn Monroe has seen a slow, steady increase in references even decades following her death.

References to “God” have declined steadily since its peak usage in the 1840s, which likely comes as no surprise. More interestingly, references to “men” far outpaced women all through the 1800s and 1900s … until the 1980s when the two were at parity. And by 2000, references to women surpass those of men.

When evaluating emotional concepts, the researchers have found that concepts like “empathy” and “self esteem” have exploded since the 1940s and 1950s … while those of “will power,” “self control” and “prudence” have all declined.

Commenting on the importance of this academic research, Mark Liberman, a computational linguist at the University of Pennsylvania, said, “We see patterns in space, time and cultural context on a scale a million times greater than in the past.”

It turns out that Google’s digital database of books is but a small fraction of the total number of volumes published since the invention of the printing press; that figure has been estimated at ~129 million. But Google’s 5 million+ books are giving us a much more precise view of trends than what’s ever been possible before.

And an interesting ancillary finding of the research is realizing the number of completely new words that have come into use in the English language. It turns out that more than 500,000 new English words that have made their “debut” since 1950.

Google is making this data available at a time when it continues to face criticism about its online library endeavor. The initiative has faced copyright disputes, lawsuits and charges that Google is attempting to create an “information monopoly” (some of which have been sort of settled). But over the long haul, I think it’s a pretty safe bet that people will view the pluses as outweighing the minuses in Google’s library project.

What’s Happening with Web Search Behaviors?

Search EnginesMore than 460 million searches are performed every day on the Internet by U.S. consumers. A new report titled 2010 SERP Insights Study from Performics, an arm of Publicis Groupe, gives us interesting clues as to what’s happening in the world of web search these days.

The survey, fielded by Lancaster, PA-based ROI Research, queried 500 U.S. consumers who use a search engine at least once per week, found that people who search the Internet regularly are a persistent lot.

Nine out of ten respondents reported that they will modify their search and try again if they aren’t successful in their quest. Nearly as many will try an alternate search engine if they don’t succeed.

As for search engine preference, despite earnest efforts recently to knock Google down a notch or two, it remains fully ensconced on the top perch; three-fourths of the respondents in this survey identify Google as their primary search engine. Moreover, Google users are less likely to stray from their primary search engine and try elsewhere.

But interestingly, Google is the “search engine of choice” for seasoned searchers more than it is for newbies. The Performics study found that Google is the leading search engine for only ~57% of novice users, whereas Yahoo does much better among novices than regular users (~36% versus ~18% overall).

What about Bing? It’s continuing to look pretty weak across the board, with only ~7% preferring Bing.

The Performics 2010 study gives us a clear indication as to what searchers are typically seeking when they use search engines:

 Find a specific manufacturer or product web site: ~83%
 Gather information before making a purchase online: ~80%
 Find the best price for a product or service: ~78%
 Learn more about a product or service after seeing an ad elsewhere: ~78%
 Gather information before purchasing in-store or via a catalog: ~76%
 Find a location for purchasing a produce offline: ~74%
 Find coupons, specials, or sales: ~63%

As for what types of listings are more likely to attract clickthroughs, brand visibility on the search engine results page turns out to be more important than you might think. Here’s how respondents rated the likelihood to click on a search result:

 … If it includes the exact words searched for: ~88%
 … If it includes an image: ~53%
 … If the brand appears multiple times on the SERP: ~48%
 … If it includes a video: ~26%

The takeaway message here: Spend more energy on achieving multiple high SERP rankings than in creating catchy video content!

And what about paid or sponsored links – the program that’s contributing so much to Google’s sky-high stock price? As more searchers come to understand the difference between paid and “natural” search rankings … fewer are drawn to them. While over 90% of the respondents in this research study reported that they have ever clicked on paid sponsored listings, only about one in five of them do so on a frequent basis.