Exposed: Today's biggest market research sampling practice flaws
27 May 2015
Pondering Panda's new CEO Diane Gantz hit the new role running with a talk at the recent annual SAMRA conference, based on her paper titled: Sampling - The Acid Test...
Diane Gantz became CEO of Pondering Panda in May 2015. And it's been a productive first month, as she's already made a presentation at the Southern African Marketing Research Association's (SAMRA) annual conference at the Cathedral Peak Hotel in the Drakensberg.
In her talk, she spoke of unrepresentative samples, which lead to invalid results being delivered to clients and demonstrated how coverage bias and non-response can skew the results of surveys, leading to incorrect findings.
It's an intriguing topic for anyone with an interest in research or marketing in general as the implications are widespread. I chatted to Gantz to find out more...
1. What was this year's SAMRA conference attendance like?Gantz: I don't know exactly how many delegates attended the conference, but it was well over a 100. It did seem like fewer clients attended this year, possibly due to the isolated location. What was great about the location, however, was that one could really mingle and get to know fellow industry players.
SAMRA conference attendees exploring the surrounding area 2. What was the general response from attendees at the conference - were they engaged?Gantz: Yes, for most part they were engaged and there were quite a few floor discussions about some of the issues that were raised.
3. That's a great start. I'd like to give our readers some insight into what you shared in your talk. Please elaborate on non-response as the major market research sampling practice flaw at the moment...Gantz: Well, it is not a new problem, but rather an ever-increasing problem... a problem that's been swept under the carpet for a while now.
1. What was this year's SAMRA conference attendance like?Gantz: I don't know exactly how many delegates attended the conference, but it was well over a 100. It did seem like fewer clients attended this year, possibly due to the isolated location. What was great about the location, however, was that one could really mingle and get to know fellow industry players.
SAMRA conference attendees exploring the surrounding area
2. What was the general response from attendees at the conference - were they engaged?Gantz: Yes, for most part they were engaged and there were quite a few floor discussions about some of the issues that were raised.
3. That's a great start. I'd like to give our readers some insight into what you shared in your talk. Please elaborate on non-response as the major market research sampling practice flaw at the moment...Gantz: Well, it is not a new problem, but rather an ever-increasing problem... a problem that's been swept under the carpet for a while now.
2. What was the general response from attendees at the conference - were they engaged?Gantz: Yes, for most part they were engaged and there were quite a few floor discussions about some of the issues that were raised.
3. That's a great start. I'd like to give our readers some insight into what you shared in your talk. Please elaborate on non-response as the major market research sampling practice flaw at the moment...Gantz: Well, it is not a new problem, but rather an ever-increasing problem... a problem that's been swept under the carpet for a while now.
3. That's a great start. I'd like to give our readers some insight into what you shared in your talk. Please elaborate on non-response as the major market research sampling practice flaw at the moment...Gantz: Well, it is not a new problem, but rather an ever-increasing problem... a problem that's been swept under the carpet for a while now.
Consumer attitudes are changing and response rates to surveys are therefore declining significantly. This means interviewers are experiencing very high levels of refusal to participate in research. There are many factors driving consumers to opt out or decline to participate in research surveys, which I'll address in answering the next question below. But another reason for the ever-increasing problem of non-response is the technology-driven era we're living in. People opt in to surveys voluntarily and most surveys are self-complete. It is therefore very easy to opt out of a survey if it is too long or too boring. With digital surveys, the problem of coverage bias also exists. This means that because each sample platform has its own inherent skews to one or the other demographic group, you can't reach the entire population. Due to the increase in non-response and coverage bias in both digital and non-digital surveys, it has therefore become extremely difficult, if not impossible, to achieve a truly representative sample.
4. Let's delve deeper then into the reasons people don't take part in surveys these days.Gantz: These include time pressures and security concerns. Consumers are empowered and have the ability to opt out of surveys quickly, especially as many surveys are now self-completed. Consumers also caught onto the idea that their time and opinions are valuable and they want to be incentivised. If an incentive is not offered, they might just refuse to participate. Lack of interest in topics or boring topics, as well as long surveys also have an impact on response rates.
5. OK then: Tell us more about the need for the Pondering Panda acid test.Gantz: Because sampling best 'old' practices, such as random sampling, are no longer achieving the desired samples due to the problem of non-response and coverage bias, the industry needs to find ways of checking whether the sampling frame yielded accurate and reliable data. The acid test is proposed as one such test for checking data validity and reliability. If the data compares well to other reliable external sources of data, such as AMPS data or IEC publicly released data, it means the sampling frame was sound and therefore resulted in valid results.
6. Makes sense. How have new technologies changed the way we do research?
Diane Gantz, new CEO of Pondering PandaGantz: New technologies have resulted in researchers having the ability to conduct research 'in-the-moment' and while consumers are 'on-the-go'.
4. Let's delve deeper then into the reasons people don't take part in surveys these days.Gantz: These include time pressures and security concerns. Consumers are empowered and have the ability to opt out of surveys quickly, especially as many surveys are now self-completed. Consumers also caught onto the idea that their time and opinions are valuable and they want to be incentivised. If an incentive is not offered, they might just refuse to participate. Lack of interest in topics or boring topics, as well as long surveys also have an impact on response rates.
5. OK then: Tell us more about the need for the Pondering Panda acid test.Gantz: Because sampling best 'old' practices, such as random sampling, are no longer achieving the desired samples due to the problem of non-response and coverage bias, the industry needs to find ways of checking whether the sampling frame yielded accurate and reliable data. The acid test is proposed as one such test for checking data validity and reliability. If the data compares well to other reliable external sources of data, such as AMPS data or IEC publicly released data, it means the sampling frame was sound and therefore resulted in valid results.
6. Makes sense. How have new technologies changed the way we do research?Diane Gantz, new CEO of Pondering PandaGantz: New technologies have resulted in researchers having the ability to conduct research 'in-the-moment' and while consumers are 'on-the-go'.
5. OK then: Tell us more about the need for the Pondering Panda acid test.Gantz: Because sampling best 'old' practices, such as random sampling, are no longer achieving the desired samples due to the problem of non-response and coverage bias, the industry needs to find ways of checking whether the sampling frame yielded accurate and reliable data. The acid test is proposed as one such test for checking data validity and reliability. If the data compares well to other reliable external sources of data, such as AMPS data or IEC publicly released data, it means the sampling frame was sound and therefore resulted in valid results.
6. Makes sense. How have new technologies changed the way we do research?Diane Gantz, new CEO of Pondering PandaGantz: New technologies have resulted in researchers having the ability to conduct research 'in-the-moment' and while consumers are 'on-the-go'.
6. Makes sense. How have new technologies changed the way we do research?Diane Gantz, new CEO of Pondering Panda
Gantz: New technologies have resulted in researchers having the ability to conduct research 'in-the-moment' and while consumers are 'on-the-go'.
Digital surveys now follow a device-agnostic approach, where technology enables researchers to engage with consumers on the device they are most comfortable with - whether it is a feature phone, smartphone, desktop, laptop, tablet or other online devices.
Most digital surveys are self-complete, which means it is very easy for respondents to opt out of surveys. Tolerance towards sloppy surveys that are long and boring is very low, so researchers need to be smart and creative when it comes to research design.
7. In the recent UK elections, all the pollsters failed to predict the majority win by the Conservative Party - how could this have been avoided? Gantz: Nate Silver, a guru in predictive analytics and forecasting, has been quoted in many articles since the UK elections as having argued that the pollsters failed because of their inability to achieve representative samples, which in turn led to invalid data. He also mentioned this is an increasing problem worldwide, which points to the problem of increasing refusal rates to participate in surveys or non-response, as well as coverage bias, due to online polls being conducted. Another contributing factor to the flawed prediction was that many believed people didn't want to indicate to pollsters that they were going to cast a 'politically incorrect' vote.
8. Really interesting. What do researchers need to do in the new world of digital research to mitigate this risk and balance biases, based on Pondering Panda's example? Gantz: A combination of measures can be taken to reduce the impact of non-response and coverage bias. Response rates can be boosted by having shorter questionnaires; a gamified approach to research, to make it less boring; and by experimenting with different ways of incentivising respondents to participate in research - by offering them content, feedback on research results or appropriate airtime incentives. In order to reduce coverage bias significantly, Pondering Panda is now sampling across multiple bases as we've found that each individual base has its own inherent skews, with certain demographic groups over- or under-represented in each base - whether it is a social media platform, online community or panel, mobile community or mobile list provider. If a 'national representative study' is required, it therefore becomes necessary to sample across multiple bases. If a very specific target group is of interest, clients need to select the base or combination of bases that is most representative of that demographic target group.
7. In the recent UK elections, all the pollsters failed to predict the majority win by the Conservative Party - how could this have been avoided? Gantz: Nate Silver, a guru in predictive analytics and forecasting, has been quoted in many articles since the UK elections as having argued that the pollsters failed because of their inability to achieve representative samples, which in turn led to invalid data. He also mentioned this is an increasing problem worldwide, which points to the problem of increasing refusal rates to participate in surveys or non-response, as well as coverage bias, due to online polls being conducted. Another contributing factor to the flawed prediction was that many believed people didn't want to indicate to pollsters that they were going to cast a 'politically incorrect' vote.
8. Really interesting. What do researchers need to do in the new world of digital research to mitigate this risk and balance biases, based on Pondering Panda's example? Gantz: A combination of measures can be taken to reduce the impact of non-response and coverage bias. Response rates can be boosted by having shorter questionnaires; a gamified approach to research, to make it less boring; and by experimenting with different ways of incentivising respondents to participate in research - by offering them content, feedback on research results or appropriate airtime incentives. In order to reduce coverage bias significantly, Pondering Panda is now sampling across multiple bases as we've found that each individual base has its own inherent skews, with certain demographic groups over- or under-represented in each base - whether it is a social media platform, online community or panel, mobile community or mobile list provider. If a 'national representative study' is required, it therefore becomes necessary to sample across multiple bases. If a very specific target group is of interest, clients need to select the base or combination of bases that is most representative of that demographic target group.
8. Really interesting. What do researchers need to do in the new world of digital research to mitigate this risk and balance biases, based on Pondering Panda's example? Gantz: A combination of measures can be taken to reduce the impact of non-response and coverage bias. Response rates can be boosted by having shorter questionnaires; a gamified approach to research, to make it less boring; and by experimenting with different ways of incentivising respondents to participate in research - by offering them content, feedback on research results or appropriate airtime incentives. In order to reduce coverage bias significantly, Pondering Panda is now sampling across multiple bases as we've found that each individual base has its own inherent skews, with certain demographic groups over- or under-represented in each base - whether it is a social media platform, online community or panel, mobile community or mobile list provider. If a 'national representative study' is required, it therefore becomes necessary to sample across multiple bases. If a very specific target group is of interest, clients need to select the base or combination of bases that is most representative of that demographic target group.
Lots to think about. For more information on Pondering Panda, follow their Twitter account.
About Leigh Andrews
Leigh Andrews AKA the #MilkshakeQueen, is former Editor-in-Chief: Marketing & Media at Bizcommunity.com, with a passion for issues of diversity, inclusion and equality, and of course, gourmet food and drinks! She can be reached on Twitter at @Leigh_Andrews.
