Why most quantitative research is inefficient and wasteful

Luca Bertolli © 123RF.com

He elaborated with the example of random probability sampling. This requires that every item in a defined universe has a known probability of being selected and then completing the survey. However, completing a survey is a voluntary action and therefore cannot be a known proportion. The proportion of people who refuse to complete is rarely reported, not only because it is a biased estimate.

Broom continued that attempting the impossible like this is expensive, inefficient and wasteful. The most common alternative is a quota sample. This is much less expensive and services most purposes adequately. The only known advantage of a probability sample over a quota sample is that probability theory helps you understand how wrong you are likely to be in using the data.

Broom adds that a third form of research sampling has gained in popularity over the past eight years, namely sampling from a panel. In fact, panels are a very old form, and are extensively used for large and important studies. TV audience measurement and retail auditing are two of the most obvious examples, with Nielsen dominating the market.

Comparing cost per data point of the three sampling theologies would show that panels are the least wasteful and most efficient methodology. However, panels are also most prone to false research and are most subject to caveat emptor!

Research

Where is marketing research going?

mike broom  3 May 2016

According to Broom, it’s the same with advertising spend. Just like quantitative research, it can be wasted. I asked him to elaborate on this. "Every marketing student has heard of the famous phrase ‘I know that half my advertising spend is wasted, I just don’t know which half’. This is attributed to a variety of sources – my preference is John Wanamaker, an American department store owner in the early 1900s," he says. "To this day, nobody knows how to measure, let alone determine what the percentage is. The paper presented at Pamro sets out a new method of measurement that can help with evidence-based decision-making in the process of both creating and deploying advertising."

I asked Broom to elaborate on a few more points from his paper...

Research

A new measurement paradigm that is ideal for ad testing

mike broom  23 Jun 2017

What is the "fuzz" you refer to in your paper and how do you measure it?
‘Fuzz’ refers to the opacity of information, modelling and decision-making that exists in all decision making. Showing how it can be measured, helps us understand how it can be managed.

We measure it by testing people's responses to our questions against our proposed UMA – ‘Universal Marketing Assumption’ – which states that ‘people do not make random brand purchase decisions all the time’. Ehrenberg showed that there is strong evidence of stochastic brand choice, but that it was the exception. We measure the extent to which a person has a clear, logical and valid perception of a category.

How does the removal of this "fuzz" enable better media planning?
By knowing how clear thinking and understanding vs a superficial relationship in a category is distributed across the population, we can harness the strengths of different media and executions in media. A ‘Maven’ for example, wouldn’t need information from advertising as much as they would intimidate ‘inside info’ – so, digital may be more effective than TV, while acolytes to the category may need editorial in magazines.

You spoke about the consistency test that was established, the rating scale that was invented, and a further measurement – speed of answering the questions on the questionnaire. Please elaborate on how this all works.
There are no right or wrong answers. The consistency is defined as the extent to which each person provides answers that are logically consistent with each other.

Each answer is binary choice and the rating scale comprises two metrics. Firstly, how easy the respondent found the question to answer. Secondly, the amount of time taken per respondent over a group of such questions. These ideas are reflected in the work of Kahneman, who argues for a System 1 (intuitive), and System 2 (cognitive) mode of thinking, with System 1 being more applicable in advertising and brand choice.

You also stated that there are six components that make a successful ad. What are these components?

In the 1970s, it was common cause that attitude to the advertisement (denoted as AAd) was a strong predictor of advertising effectiveness. It was not until 1983 that the first explanation of what was meant by AAd emerged. We have interpreted the literature by postulating six components in three groups:

• Residual attitude (attitude to the brand the company)
• Personal attitudes (Attitude to advertising and to affective intent to the brand)
• Functional attributes (Information and entertainment)

Pilot and operational results have resulting in better understanding of this and have norms in relation to different categories.