Domestic Affairs

Substandard surveys are worse than no surveys

By Patrick Mattimore (chinadaily.com.cn)
Updated: 2010-08-18 13:38

Newspapers increasingly rely upon surveys to inform readers. In addition to outsourcing survey work to professional organizations, newspapers should alert the public to some of the surveys' inherent limitations.

Generally, surveys are representative samplings of larger populations of randomly selected individuals and have built in margins of error. A good survey will have a margin of error of 3% or less. That means that in 95 cases out of 100, if we were to conduct the same survey with different representative samples, the results would be within 3% (plus or minus) of our initial survey.

While procedures for statistically collecting data are fairly well-established, a more serious potential source of inaccuracy in public opinion surveying is the content of the survey itself. Specifically, the wording of questions, the order in which they are asked, and the response alternatives that are made available to respondents can all influence the results of public opinion surveys.

According to a Gallup paper on the topic, question wording is probably the greatest source of bias and error in the data, followed by question order. Writing a clear, unbiased question is an art and requires extensive knowledge about collecting information.

There are some steps news sources can take to improve survey reporting.

It is helpful if the news stories reporting results of surveys contain links to the survey questions so that interested individuals can evaluate and critique those questions and survey methodology. When possible, a newspaper should include specific questions asked in the survey as "China Daily" did with a survey about China and Japan published on Monday.

Newspapers should invite experts' and laypersons' analyses of the data and, when appropriate, make those analyses available to the public.

Surveys should disclose who conducted the polling and any outside funding received. Newspapers have a duty to alert the public as to possible biases of funders.

Press releases about surveys should detail limits of the particular research and headlines should not oversell results. For example, if there has been a low response rate, the release might urge the public to interpret the results with greater caution.

Surveys are tools that can reveal, inform, and shape public opinion. The utility of surveys depends largely upon the expertise of the pollsters and the degree to which the public trusts the sources reporting results of those surveys.

About commentator: Patrick Mattimore is a fellow at the Institute for Analytic Journalism and an adjunct professor at Tsinghua/Temple Law School LLM Program in Beijing