The City of Abbotsford is wasting its money by using surveys that aren’t statistically valid and which can’t be said to accurately gauge residents’ opinions, says a University of the Fraser Valley math professor.
Last week, the city released the results of its first quality-of-life survey, which aimed to measure residents’ feelings about the city, their life in the community and their municipal government. The city also recently rolled out a new website to collect feedback from the public, including through such self-selected surveys.
The survey was undertaken by a private company that says it works to build “long-term community engagement in municipalities,” and the results were meant to set a baseline for future surveys, so the city could tell if certain areas were getting better or worse. In a press release, the city claimed that “based on the key demographics collected, it can be said that the survey findings fairly represent the population of Abbotsford.”
RELATED: High marks for Abbotsford’s fire department in first survey
RELATED: Abbotsford launches new website to get feedback from public
But UFV math and statistics professor Gillian Mimmack said the quality-of-life survey results aren’t useful at all. The same, she said, goes for other city surveys that rely on the opinions of “self-selected” voluntary respondents, rather than randomly selected community members.
“The whole survey is not valid,” Mimmack said of the quality-of-life survey. “You can’t just do a voluntary response survey, which gives you a biased sample, and say, ‘Here you go.’ It’s a waste of money, waste of time, waste of everything.”
When it released the results, the city reported that “residents have a strong sense of community and residents are positive regarding their Quality of Life.” Four in five had reported they were “living close to their best possible life,” and half said they had a “strong sense of belonging to the community.”
The problem is that certain types of people will inevitably be under-represented in a self-selected survey and the feelings of those under-represented people “cannot be guessed.”
Self-selected surveys don’t account for the fact that some groups of people are less likely to hear about the survey. And online surveys predictably over-sample the types of people who use the internet the most and under-count those groups less likely to be online.
Demographic data collected by the city includes respondents’ length of time in Abbotsford, their region of residency and their age. It didn’t ask about language or cultural backgrounds. Only age information was released, and it showed respondents were more middle-aged than the city as a whole. Seniors were also under-represented. It’s also not clear whether surveys were available in languages other than English; one-third of Abbotsford’s population has a mother tongue other than English.
“The only way to obtain a representative sample is to do some sort of random sampling,” Mimmack said.
The City of Abbotsford isn’t alone in relying on online, self-selected surveys. The previous Conservative government’s decision to make the 2011 long-form census survey voluntary led to the resignation of the head of Statistics Canada and rendered much of the resulting data untrustworthy by health bodies and other institutions. Mimmack added that large polls by major media companies during the 2016 election also relied on populations that weren’t randomly selected.
Polling random samples by phone would cost money, but would give the city a truly accurate picture of residents’ feelings, Mimmack said. A random sample of 267 people would give an “adequate” margin of error of six per cent. The city’s quality-of-life survey drew responses from more than 1,400 people, but voluntary surveys cannot be said to have a margin of error.
“Large, bad samples are just garbage,” Mimmack said. “It doesn’t give them a good idea of what’s going on.”
The city said this year’s quality-of- life survey will set a baseline. But comparing future results to this year’s survey may be misleading because shifts in the makeup of respondents can have a large impact on results.
“They need to start doing it properly and they’ll get a real view of what people think,” Mimmack said
Mimmack said the area has many statisticians who would be able to help the city conduct a survey that uses a random sample.
The News was not able to speak to the operator of the company that conducted the survey by press time.. A written response from the City of Abbotsford was not received by press deadline.
In 2010, the city contracted Ipsos Reid, a respected pollster, to conduct a survey of 300 randomly selected citizens over the age of 18 on local issues. The survey cost $11,300.
A 2006 report by the city’s then-IT head on the availability of online survey technology cautioned: “The information is useful, but as the responses are not part of a randomly selected representative sample of the population, they are not generalizable to the general population.”
@ty_olsen
tolsen@abbynews.com
Like us on Facebook and follow us on Twitter.