In survey research, money talks … but to what degree?

For anyone who has attempted to survey consumers and businesses, it’s pretty universally understood that in order to boost the response rate, you need to give people a “WIIFM” reason to respond.

And that WIIFM incentive is often money. But what kind of monetary incentive works best these days, considering all of the different ways that people are being asked to participate in surveys?

One thing’s for sure: the trend data on response rates isn’t encouraging.  In 1997, the average response rate on telephone surveys was around 36%.  As of 2012, the percentage had nose-dived to just 9%.

It can’t have gotten any better in the five years since.

Recently, the Gallup organization set about to determine response rate dynamics in relationship to the types of monetary incentives offered. To do this, Gallup took the alumni listing from a major American university and deployed online surveys to three target groups of names drawn from it.

Each group was made up of randomly selected names, and each group received the exact same survey. The only difference was in in the incentive offered for recipients to respond to the survey:

  • Group A: 10,000 targeted people received no monetary incentive
  • Group B: 1,000 targeted people were promised a $5 gift card after completing the survey (post-paid incentive)
  • Group C: 1,000 targeted people received a gift card as part of the survey invitation (pre-paid incentive)

The Gallup test revealed that, as expected, offering a monetary incentive had a significant impact on the survey response rate:

  • Group A: 13% response rate
  • Group B: 20% response rate
  • Group C: 19% response rate

But perhaps more interestingly, the results suggest that a pre-paid incentive isn’t quite as strong as offering a monetary reward that comes after filling out the survey. Albeit, the results are very similar, so no definitive conclusions can be drawn.

What is clear, though, is that offering a monetary incentive of some kind does dramatically improve survey results – to the tune of ~50% higher.

Moreover, the Gallup research found no behavioral differences between income groups, suggesting that the “psychology” of being offered a token of appreciation for the survey-taker’s time is something universally appreciated, rather than it being tied to particular respondent characteristics like financial status.

Additional information about the Gallup research can be accessed here.

Quick-change artistry: Masculinity goes from “alpha-male” to “alta-male” inside of a generation.

Alpha-male: Venezuelan actor Alejandro Nones
Alpha-male: Venezuelan/Mexican actor Alejandro Nones

Many people contend that changes in society are driven by many influences – not least movies and music. Certainly, the popular arts reflect the current culture, but they also drive its evolution.

This view was underscored recently in the results of field research conducted by a British- and Singapore-based survey firm Join the Dots for Dennis Publishing, which has just launched Coach, a magazine in the U.K.

The magazine’s audience consists of men who are committed to lifestyles that make themselves “healthier, fitter and happier.”

The research aimed to figure out what are today’s characteristics of being “male.” An in-depth qualitative focus group session with men aged 22 to 60 helped establish the set of questions that was then administered in a quantitative survey of ~1,000 respondents (including some women as well as men) between the ages of 25 and 54 years old.  The survey sample represented a diverse mix of family status, sexual preferences, incomes, professions and interests.

The survey questions focused on the habits and aspirations of men … and the results showed how far we’ve come from the heydays of the “alpha-male” barely 25 years ago.

The researchers contrasted good and not-go-good alpha-male stereotypes (self-absorbed … unwilling or unable to talk about insecurities or vulnerabilities) with a new persona they dubbed the “alta-male.”

The alta-male is a man who values work/life balance and finds personal fulfillment as much in self-improvement as in material wealth.

Additionally, the alta-male tends to reject male role models from earlier generations, instead opting to establish their own identity based on a myriad of diverse influences.

Of course, it’s one thing to aspire to these goals and quite another to actually attain them. The study found that two-thirds of the respondents are finding it difficult to achieve the satisfactory work/life balance they desire.

On the other hand, alta-males tend to be more adaptable, and they’re willing to embrace uncertainty more than the alpha-males of yore.

Even more strongly, alta-males are seekers of experiences, which they value over “mere money” – despite recognizing that it takes money to partake in many such life experiences.

More of an alta-male: American businessman Phillip Nones

Perhaps most surprising, the study found little difference in perspectives between older and younger male respondents.

It turns out that older men are just as likely to have an “alta-male” attitude towards life.  So clearly, the culture has been rubbing off on them, too.

From my own personal standpoint (as someone whose been around the track quite a few times over the decades), I sense a similar shift in my own personal perspectives as well.

What about the rest of you?

Is an online survey always the “slam-dunk” methodology for field research projects?

srIt’s been quite a long time since I’ve received an invitation to participate in a telephone research survey. And postal mail surveys?  I haven’t been asked to participate in one of those in eons.

It isn’t hard to understand why. Online surveys have become the “default” option for quantitative research.  Not only has digital shaped the way people interact, communicate and shop, online research has plenty of advantages over the more traditional survey methods.  It’s cheaper … it’s faster … it can be quite engaging … and it’s in line with modern behavior.

But being involved in market research in my business, I’m also finding that online surveys have their drawbacks — and it’s becoming more evident with each passing year.

The biggest problem? We’ve seen online survey participation rates crater in recent years as more “stuff” crams people’s inboxes.

When you’re forced to deploy survey invitations to thousands of e-contacts in order to obtain a few hundred usable responses, that’s a symptom of a pretty big problem.

And it leads to another potential concern: Will the respondent pool be representative of the required population?

We’ve known for a long time that certain groups tend to be underrepresented in terms of Internet or digital engagement. And now … to that we can add those people who suffer from “inbox overload.”

In the B-to-B world especially, it isn’t uncommon for a manager to receive 150+ e-mails in a single business day.  Not surprisingly, the great majority of them are trashed without any sort of recipient engagement whatsoever.

… And there goes your research invitation and online survey link.

Online spamming is also contributing to lower online survey participation rates as more people become concerned about the potential dangers of spam mail, thus hesitating to engage with unsolicited e-mails.

On the other side of the coin are the people who have become “professional respondents.” As the financial incentives to participate in surveys have become more lucrative, some people are in the business of survey-taking as revenue-generating proposition.

One wonders how “engaged” these people really are as survey takers — or if they care at all about the topics being studied.  “A mile wide and an inch deep” is more their style.

pst

Ironically, the sum total of these concerns seems to be making phone surveys (the CATI kind — computer-assisted telephone interviewing) the “new-old” alternative to online surveys.

CATI surveys are more expensive and more time-consuming. But in some cases, they may represent the difference between the success or failure of a research endeavor.

What’s more, employing a phone methodology can provide better control over the survey process, along with greater depth and “nuances” regarding the insights gleaned.

Am I going to be recommending CATI telephone survey methodologies to our clients going forward? In most cases, the overall economics — the price-to-value equation — still heavily favors online survey methodologies.

But the gap between them is narrowing … and I can easily envision at least some instances where a return to the classic methodologies of the past may be just what’s needed in the present.

Where Outside Suppliers of Business Services Fall Down on the Job …

Quirk's Corporate Research ReportQuirk’s Marketing Research Review is a periodical I’ve enjoyed reading for three decades or more.  Unlike the articles that appear in other research-related publications that are more “scholarly” and theoretical,  I find the articles in Quirk’s to be chockfull of insights, while at the same time being “efficiently practical” and easy to digest.

Recently, the magazine published findings from its second annual Quirk’s Corporate Research Report, designed to give corporate researchers an in-depth look into their world.

As part of the research-gathering process for the report, Quirk’s conducted a field survey covering budgets, outsourcing, research techniques in use and under consideration, how research findings are reported inside organizations and, last but not least, the experiences researchers have had when working with outside vendors.

When asked by Quirk’s to state what are the main problem areas when research vendors have come up short on a project, these eight factors were cited by respondents most often:

  • The vendor over-promised and under-delivered: ~56% of respondents mentioned
  • The project was handled by low-level staff: ~51%
  • Vendor failed to take time to understand the client’s business: ~50%
  • Vendor had poor communications: ~39%
  • Vendor failed to take time to understand the project’s needs: ~36%
  • Data integrity issues: ~35%
  • Vendor missed deadlines: ~35%
  • Tools/methodologies that the vendor suggested weren’t right for the project: ~14%

Notice how the most pervasive issues have less to do with the inherent quality of the research product being delivered, and more to do with how the vendor interfaces with and communicates with the companies they support.

The above behaviors represent challenges associated with conducting research projects. But I contend that they apply equally well to providers of other types of business and corporate services, whether they’re ERP or IT projects, website development projects, CRM implementation, SEM/SEO programs, media campaigns, PR initiatives … even IPOs, capital campaigns and the like.

Which of these shortcomings do you find to be most prevalent in your dealings with outside service providers — and what have you done about them? Please share any insights you may have with other readers here.

Misusing Marketing Research: There’s a Saying for That

How to Lie with Statistics by Darrell Huff (1954)
How to Lie with Statistics, Darrell Huff’s business classic, first published in 1954.

Personally, I have respect for marketing research as a discipline.  I think most business decisions are better when they’re backed by the power of marketing research.

Still, I recognize that research can also be used in misleading or otherwise improper ways.

Even worse, research results can be contorted to justify business decisions that have been predetermined.  All too often, “How can we produce results that justify our position?” is the impetus behind a research initiative.

It’s that “dirty little secret” of research that was brought to light decades ago in Darrell Huff’s business classic, How to Lie with Statistics.  First published in 1954, this book been published in countless editions and remains in print even today, 60 years later.

Quirk's Marketing Research ReviewRecently, Dan Quirk of Quirk’s Marketing Research Review, the American research industry’s leading practicum publication, asked subscribers to share their favorite research-related quotes — ones that point to the folly that can be part of the discipline at times.

Some of the reader contributions are great — and they certainly point to the downsides of the research field.  Consider these bon mots:

“Science is built of facts the way a house is built of bricks … but an accumulation of facts is no more science than a pile of bricks is a house.”  (attributed to Henri Poincaré)

“Don’t let the facts get in the way of the truth.”

“When research walks on the field, judgment does not walk off.”  (attributed to Richard Kampe)

“Don’t theorize before one has data:  One begins to twist facts to suit theories, instead of theories to suit facts.”  (attributed to Sir Arthur Conan Doyle)

“Precise forecasts masquerade as accurate ones.”  (attributed to Nate Silver)

“If you torture a data set long enough … it will confess.”

“There are three kinds of lies: lies, damned lies, and statistics.”  (attributed to Mark Twain)

“Statistics can be misleading; the average human has one breast and one testicle.”

“A little nonsense now and then is relished by the wisest men. (attributed to Roald Dahl)

And this one, which ties everything up in a neat little bow:  “No research is better than bad research.”

If you have other memorable research quotes to add to the list, please share them with other readers here.  It’ll be good for a chuckle at least!

Fake online product reviews: How pervasive are they?

Fake reviewsThink about those reviews that mean so much to you when considering whether to purchase a particular product or a service …

It could be that the comments you’re reading are bogus – or at least not based on the reviewer’s first-hand experience.

An online survey of nearly 1,200 U.S. adults age 18 and older, conducted by marketing research firm YouGov in January 2014, found that more than one in five respondents admitted to having posted online reviews about products or services they hadn’t actually bought or used.

The percentage is somewhat higher for men (~23%) than it is for women (~17%).

Why do people post reviews or comments on products and services they haven’t tried?  Here’s what the survey respondents reported:

  • “Just felt like it”:  ~32% gave this reason
  • “Didn’t like the idea of the product”:  ~22%
  • “Didn’t like the manufacturer”:  ~19%

These stats might suggest that there are more “negative” reviews being posted online than what reflects the actual experience with the product or service.

But the YouGov survey also found that far more people leave good reviews than bad ones:

  • ~57% have left a mixed review
  • ~54% have left a good review
  • Only ~21% have ever left a bad review

What drives someone to leave a bad review?  The #1 reason is obvious … but the #2 reason might surprise you.  And the #3 reason is just mercenary:

  • ~88% want to warn others about a disappointing product or service
  • ~23% believe that venting their frustrations will leave them feeling less angry
  • ~21% are hoping to get a refund or some other monetary consideration from the company in question

The veracity of online reviews is important because the vast majority of adult consumers check them before deciding to purchase a product or service.

This YouGov survey is no different:  It found that ~79% consult reviews at least sometimes … and ~26% reported that they “always” check reviews before buying a product or service.

FakeryThe YouGov report comes hard on the heels of a Virginia lawsuit wherein a carpet cleaning service charged online review website Yelp with publishing negative reviews posted by people who had never been customers of the store.  The cleaning service claimed that the negative reviews had hurt its business.

In that case, a judge ordered Yelp to reveal the identities of the seven “anonymous” reviewers — who I’m sure never thought their “unidentified antics” would ultimately be revealed for all the world to see.

It may just be that posting a “faux” review has now become a little riskier.

People may think twice now before engaging in their little mischief.  I’m sure most of them can think of a lot better things to do than to be hauled into court for an alleged infraction like that — or at the very least, having their name brought into the legal proceedings.

Marketing Fail? Too Many Mobile Apps are Deleted within Days of Downloading

Mobile appsHere’s an interesting statistic offered up by marketing consultant Rich MeyerThree-fourths of mobile apps are deleted within three weeks of being downloaded by their users.

How can the attrition rate be so high?

According to Meyer, it’s because people decide they don’t really have a need for the apps … or they find them too difficult to use and master.

I suspect the percentage may also be so high because marketers fail to query their target audiences prior to developing apps to determine now much of a need it will be satisfying.

… Or to put it another way, to avoid falling into the trap of developing a cure for something that isn’t a disease.

Mobile App Preferences
Sources: MarketingProfs; Harris Interactive and EffectiveUI field survey, 2010.

Meyer believes part of the dynamic at work is a knee-jerk “bias for action” as the marketing playing field shifts endlessly.

“It’s called ‘do it’ because everyone else is doing it, and it results in not only bad marketing, but in turned off consumers and customers,” he maintains.

Questions as simple as “What would you like to see in a mobile app?” … or testing an app concept with a sample of potential users before spending the effort and energy to produce it would be good places to start.

Marketers can use the research findings to adjust the proposed design of an app — or to trash it altogether and come up with an alternative one that actually meets a need.

If more companies did this, perhaps the 75% deletion rate for mobile apps would cease to be so flat-out dismal.