Are we ready to take responsibility for SAARF's AMPS data accuracy?

SAARF's bi-annual all media and products survey or AMPS is intended to provide the media, marketing and advertising industries with all-important data on media consumption, demographics and products, brands and services. However, for a long time now this data has been scoffed at, seen as unreliable and on shaky ground due to much publicised financial issues. The presentation of SAARF's 2014a release, which has been available since 2 October, attempted to plug the hole.

Etienne van den Berg, who has been taking care of SAARF for most of 2014 since his appointment as interim CEO, welcomed attendees at the Protea Hotel Fire and Ice. The morning road show presentation was broken down into three parts. First up was Sue Scott, providing a Nielsen update on AMPS 2014a, followed by Andrew Fulton of Eighty20 in an amusing and insightful presentation on how consumers use the AMPS data, ending with an update on SAARF and the AMPS roll over.

SAARF AMPS 2014a data highlights

Scott clarified that the sample size for AMPS 2014a was 25,510, pointing out that interesting developments this time around include the return of the hi-fi/music centre category, as many see their iPod docking centre seen as a music centre. Another interesting point is that the category of two cellphones per household has fallen away, as the average is now three or more. There's also been some regression in the newer LSMs, with average household income increasing to R9,997 per month, but Scott says to use some caution as this is not seen as a real increase as it's not keeping up with inflation.

I tweeted some of the stand-out points from Scott's presentation:

Interesting that #saarf2014a shows dip in male readers for Rooi Rose and in 25 to 34 age group for Your Pregnancy. Demographic shift?

- Rambling Litchi (@Leigh_Andrews) October 23, 2014

Wow. #saarf amps 2014a shows search and chat as top 2 internet activities, receive& send Please Call Me as top 2 cellphone acts among adults

- Rambling Litchi (@Leigh_Andrews) October 23, 2014

In summary, Scott took a question from the audience on the fact that ABC figures show sales of print media consistently slowing over the past five years, asking how it's then possible for the AMPS to claim newspaper and magazine readership is relatively stable as per its previous figures. Scott answered that the increase in readership shown in AMPS is due to high pass on rate of print titles, like the Daily Sun's sensationalist headlines that ensure the paper is picked up by someone else if left in a train carriage or on a taxi seat. A lag effect also exists, so the AMPS results are actually not dissimilar to those presented in the ABCs and Scott says it's possible to overlay the ABC and AMPS data.

Laughing off the data myths with Andrew Fulton of Eighty20

Andrew Fulton of Eighty20 speaking at the Protea Hotel Fire and Ice

Following a quick coffee break we heard from Andrew Fulton of Eighty20 in a presentation titled 'Sturgeon's Law and other myths about AMPS'. Fulton was tongue-in-cheek about the need to keep an open mind when it comes to this type of survey data, as AMPS is a constantly evolving survey. It's been updated each year since its 1974 launch based on stakeholder input to keep up with new developments in the media and product landscape. Added to this, there are challenges of media audience measurement worldwide due to media proliferation and fragmentation, which have resulted in changes to the survey.

But there's more to it than that. Fulton explained that as one of the owners of Eighty20, part of his job lies in making sense of data like the AMPS for his clients based on secondary and company data. Speaking on the accuracy of SAARF's AMPS data, he explained that it adds colour to other data he analyses for clients. With SAARF currently in financial turmoil, Fulton asked what the world would be like without AMPS - we'd have to draw straws over which is the most popular brand, and list the data source as 'nothing' - proving the point that AMPS data is better than no data.

At the end of the day, it's about making the data interesting. While he admits he's not an unbiased observer, Fulton says the loss of AMPS would leave a hole in revenue, and at the end of the day the data is underused. There is an overwhelming amount of poor quality data out there that crowds out the good data. For example, he points out that right now about 32% of South Africa is online, so it's impossible to find a nationally representative sample. That said, SAARF's data sample is very large compared to others with smaller data sets, and is solid as it's backed by Nielsen, featuring hundreds of brand categories, so if you look properly you can get consumer brand information on just about anything.

Be careful out there: Data gets twisted along the way

Fulton clarified that he didn't intend giving a sunshine and roses lecture, he admits that AMPS has its flaws, but not enough to simply throw the data out. Fulton recommends that with any data, you check the accuracy by actually asking people what they think, checking with experts and triangulating it. Particularly when it comes to financial, health and sexual information, remember that people give incorrect information or lie. In addition to all this, remember that you can use data to tell whatever story you want; there is always some bias or twist, so you need to take all data quoted anywhere with a pinch of salt. Fulton says the point is not about reaching the final figure as an answer but rather about whether you're asking the right questions; any data can be inaccurate as it is dated and people forget or worse, lie in giving responses, so you'll never get the absolute truth. It all depends on who you're asking and what you're extrapolating about.

"Amps is flawed" All #media research is flawed & to expect otherwise from any outcome is naïve.

http://t.co/WgMrh3OU3p via @themediaonline

- Gordon Muller (@mzansimedia) December 12, 2013

Funding and the future of SAARF... over to you

Van den Berg closed the session by saying we're at the end of 2014 now, and his aims for the year were to get out data on time and as accurate as possible, despite all politics and controversy - he feels SAARF has achieved that, despite the commotion. He pointed out that the debate is not about SAARF itself, it's about an organisation that belongs to marketers, online, broadcast and print alike, which follows best practice. It essentially gets its mandate from the industry, so if you want a change, it's up to you. He acknowledges that some members will break off and create their own, and says there's no problem with that. But you still need for central body to verify and look at credibility of the data.

Moving on to funding, Van den Berg said funding as a global issue, and that with money SAARF can provide you with whatever research you want, but for now it has to be as cost-effective as possible. He also warned not to be surprised if SAARF goes by a new name next year, and that it'll be looking for a new mandate from the industry and to relook at funding models going forward. Following a question from the audience on media owner funding, Van den Berg says it is 100% from them, proportioned as a percentage based on how much of the survey applies to each media owner. Nothing comes from the marketers now as the 1% levy from the 1990s long since fell away. This data is what sets the South African marketing and media industry apart from the rest of Africa, so someone needs to step up and ensure the research legacy doesn't crumble to dust.

Strong words, let's see if the next stretch puts these comments into play.

Click here for more on SAARF's AMPS 2014a data and to download the full report.