Abbotsford wasting time and money on ‘invalid’ surveys, UFV math prof says

Statistician says city can’t claim quality of life survey reflects opinions of Abbotsford residents

The City of Abbotsford is wasting its money by using surveys that aren’t statistically valid and which can’t be said to accurately gauge residents’ opinions, says a University of the Fraser Valley math professor.

Last week, the city released the results of its first quality-of-life survey, which aimed to measure residents’ feelings about the city, their life in the community and their municipal government. The city also recently rolled out a new website to collect feedback from the public, including through such self-selected surveys.

The survey was undertaken by a private company that says it works to build “long-term community engagement in municipalities,” and the results were meant to set a baseline for future surveys, so the city could tell if certain areas were getting better or worse. In a press release, the city claimed that “based on the key demographics collected, it can be said that the survey findings fairly represent the population of Abbotsford.”

RELATED: High marks for Abbotsford’s fire department in first survey

RELATED: Abbotsford launches new website to get feedback from public

But UFV math and statistics professor Gillian Mimmack said the quality-of-life survey results aren’t useful at all. The same, she said, goes for other city surveys that rely on the opinions of “self-selected” voluntary respondents, rather than randomly selected community members.

“The whole survey is not valid,” Mimmack said of the quality-of-life survey. “You can’t just do a voluntary response survey, which gives you a biased sample, and say, ‘Here you go.’ It’s a waste of money, waste of time, waste of everything.”

When it released the results, the city reported that “residents have a strong sense of community and residents are positive regarding their Quality of Life.” Four in five had reported they were “living close to their best possible life,” and half said they had a “strong sense of belonging to the community.”

The problem is that certain types of people will inevitably be under-represented in a self-selected survey and the feelings of those under-represented people “cannot be guessed.”

Self-selected surveys don’t account for the fact that some groups of people are less likely to hear about the survey. And online surveys predictably over-sample the types of people who use the internet the most and under-count those groups less likely to be online.

Demographic data collected by the city includes respondents’ length of time in Abbotsford, their region of residency and their age. It didn’t ask about language or cultural backgrounds. Only age information was released, and it showed respondents were more middle-aged than the city as a whole. Seniors were also under-represented. It’s also not clear whether surveys were available in languages other than English; one-third of Abbotsford’s population has a mother tongue other than English.

“The only way to obtain a representative sample is to do some sort of random sampling,” Mimmack said.

The City of Abbotsford isn’t alone in relying on online, self-selected surveys. The previous Conservative government’s decision to make the 2011 long-form census survey voluntary led to the resignation of the head of Statistics Canada and rendered much of the resulting data untrustworthy by health bodies and other institutions. Mimmack added that large polls by major media companies during the 2016 election also relied on populations that weren’t randomly selected.

Polling random samples by phone would cost money, but would give the city a truly accurate picture of residents’ feelings, Mimmack said. A random sample of 267 people would give an “adequate” margin of error of six per cent. The city’s quality-of-life survey drew responses from more than 1,400 people, but voluntary surveys cannot be said to have a margin of error.

“Large, bad samples are just garbage,” Mimmack said. “It doesn’t give them a good idea of what’s going on.”

The city said this year’s quality-of- life survey will set a baseline. But comparing future results to this year’s survey may be misleading because shifts in the makeup of respondents can have a large impact on results.

“They need to start doing it properly and they’ll get a real view of what people think,” Mimmack said

Mimmack said the area has many statisticians who would be able to help the city conduct a survey that uses a random sample.

The News was not able to speak to the operator of the company that conducted the survey by press time.. A written response from the City of Abbotsford was not received by press deadline.

In 2010, the city contracted Ipsos Reid, a respected pollster, to conduct a survey of 300 randomly selected citizens over the age of 18 on local issues. The survey cost $11,300.

A 2006 report by the city’s then-IT head on the availability of online survey technology cautioned: “The information is useful, but as the responses are not part of a randomly selected representative sample of the population, they are not generalizable to the general population.”


@ty_olsen
tolsen@abbynews.com

Like us on Facebook and follow us on Twitter.

Just Posted

UFV launches school of creative arts

New school combines theatre and visual arts

Pints n’ Pitches coming to Abbotsford

Free Oct. 16 event brings together people to foster potential business ventures

BC Children’s virtual care sites open at Abbotsford and Chilliwack hospitals

Provincial initiative allows pediatric patients to see health specialists through video

McKenzie Road shut down by police for hours last night due to mental health incident

People reportedly heard police negotiating with individual to come outside house

Historian Keith Carlson appointed Canada Research Chair at UFV

Carlson is an ethno-historian with a strong connection to the Sto:lo people in Chilliwack

VIDEO: Langley woman’s security camera records its own theft

Langley family discovers early morning grab was recorded

Canadian Snowbirds plane crashes before air show in Atlanta

Pilot lands safely after ejecting from jet

Share crash data, private insurers tell David Eby, ICBC

B.C. monopoly makes drivers retrieve their own records

B.C. VIEWS: Wolf kill, not backcountry bans, saving caribou

B.C.’s largest herds turn the corner from extinction

Vancouver Giants wrap up prairie road swing with a 4-2 win over Regina

Milos Roman paced the Langley-based Giants with two goals, goaltender David Tendeck made 34 saves

Pearson nets shootout winner as Canucks clip Flyers 3-2

Vancouver picks up second straight home win

Map on Elections Canada website sends Nanaimo-Ladysmith voters to landfill

Address for polling station correct, but Google Map address differs

Man arrested in fatal stabbing at Surrey gas station

Victim pronounced deceased at the scene; RCMP cordon off area

Most Read