The ARF (Advertising Research Foundation) is creating a committee to set new standards for online survey research. The increase of “professional respondents” and the lack of their control is a worrying trend for the industry.

The “episodes” that motivated this are basically the following:

• Last September, at a roundtable hosted by the Institute for International Research, Kim Dedeker, VP-consumer and market knowledge for Procter & Gamble Co., said two online surveys from the same provider conducted a week apart delivered diametrically opposed results.

• ComScore Networks, meanwhile, produced research last year showing that a relatively tiny number of professional respondents determine the results of online studies. ComScore’s research, conducted as part of an effort to sell a panel of nonprofessional online respondents, showed that 0.25% of the online population accounts for 32% of responses in other online surveys, while less than 5% account for more than half of the responses.

The newly formed Online Research Quality Council will hold its first meeting Sept. 10 and aim to establish a set of industry standards for evaluation by early April, when the ARF holds its annual convention in New York.

Although we believe this may be the case for the most developed markets as the US and Western Europe, we do not think anything like this is happening in Latin America. Why?

• There are not enough online research companies in the region to create “Professional respondents”

• The incentives for doing this (in Livra for example) are very very low as it is (a) virtually impossible to get chosen for a survey on purpose (b) completely impossible to make a living on the prizes we give out when completing a survey (you have to actually win the sweepstakes first)

• We NEVER reveal the profile we are after on the invitation e-mail or the screener… this is something (that we know) A LOT of companies do and definitely encourages the problems described above.

• As Latin American online research is under more scrutiny than in other markets, we were asked many times to run parallel tests with CATI or other methodologies and ALWAYS found the results very similar and consistent. Same with several tracking studies we conducted in the past.

And what about conventional research and focus groups? I personally know people who “work as a focus group participant”.

Well, this is an ongoing problem and we hope the guidelines the ARF come up with are actually properly designed to combat the issue (Did someone ever wonder what are “the correct answers” to the ESOMAR 25 questions? ).

More here…

ARF Puts Online Research Under Microscope [Advertising Age]