Abbotsford wasting time and money on ‘invalid’ surveys, UFV math prof says

Statistician says city can’t claim quality of life survey reflects opinions of Abbotsford residents

The City of Abbotsford is wasting its money by using surveys that aren’t statistically valid and which can’t be said to accurately gauge residents’ opinions, says a University of the Fraser Valley math professor.

Last week, the city released the results of its first quality-of-life survey, which aimed to measure residents’ feelings about the city, their life in the community and their municipal government. The city also recently rolled out a new website to collect feedback from the public, including through such self-selected surveys.

The survey was undertaken by a private company that says it works to build “long-term community engagement in municipalities,” and the results were meant to set a baseline for future surveys, so the city could tell if certain areas were getting better or worse. In a press release, the city claimed that “based on the key demographics collected, it can be said that the survey findings fairly represent the population of Abbotsford.”

RELATED: High marks for Abbotsford’s fire department in first survey

RELATED: Abbotsford launches new website to get feedback from public

But UFV math and statistics professor Gillian Mimmack said the quality-of-life survey results aren’t useful at all. The same, she said, goes for other city surveys that rely on the opinions of “self-selected” voluntary respondents, rather than randomly selected community members.

“The whole survey is not valid,” Mimmack said of the quality-of-life survey. “You can’t just do a voluntary response survey, which gives you a biased sample, and say, ‘Here you go.’ It’s a waste of money, waste of time, waste of everything.”

When it released the results, the city reported that “residents have a strong sense of community and residents are positive regarding their Quality of Life.” Four in five had reported they were “living close to their best possible life,” and half said they had a “strong sense of belonging to the community.”

The problem is that certain types of people will inevitably be under-represented in a self-selected survey and the feelings of those under-represented people “cannot be guessed.”

Self-selected surveys don’t account for the fact that some groups of people are less likely to hear about the survey. And online surveys predictably over-sample the types of people who use the internet the most and under-count those groups less likely to be online.

Demographic data collected by the city includes respondents’ length of time in Abbotsford, their region of residency and their age. It didn’t ask about language or cultural backgrounds. Only age information was released, and it showed respondents were more middle-aged than the city as a whole. Seniors were also under-represented. It’s also not clear whether surveys were available in languages other than English; one-third of Abbotsford’s population has a mother tongue other than English.

“The only way to obtain a representative sample is to do some sort of random sampling,” Mimmack said.

The City of Abbotsford isn’t alone in relying on online, self-selected surveys. The previous Conservative government’s decision to make the 2011 long-form census survey voluntary led to the resignation of the head of Statistics Canada and rendered much of the resulting data untrustworthy by health bodies and other institutions. Mimmack added that large polls by major media companies during the 2016 election also relied on populations that weren’t randomly selected.

Polling random samples by phone would cost money, but would give the city a truly accurate picture of residents’ feelings, Mimmack said. A random sample of 267 people would give an “adequate” margin of error of six per cent. The city’s quality-of-life survey drew responses from more than 1,400 people, but voluntary surveys cannot be said to have a margin of error.

“Large, bad samples are just garbage,” Mimmack said. “It doesn’t give them a good idea of what’s going on.”

The city said this year’s quality-of- life survey will set a baseline. But comparing future results to this year’s survey may be misleading because shifts in the makeup of respondents can have a large impact on results.

“They need to start doing it properly and they’ll get a real view of what people think,” Mimmack said

Mimmack said the area has many statisticians who would be able to help the city conduct a survey that uses a random sample.

The News was not able to speak to the operator of the company that conducted the survey by press time.. A written response from the City of Abbotsford was not received by press deadline.

In 2010, the city contracted Ipsos Reid, a respected pollster, to conduct a survey of 300 randomly selected citizens over the age of 18 on local issues. The survey cost $11,300.

A 2006 report by the city’s then-IT head on the availability of online survey technology cautioned: “The information is useful, but as the responses are not part of a randomly selected representative sample of the population, they are not generalizable to the general population.”


Like us on Facebook and follow us on Twitter.

Just Posted

Changes coming June 30 to bus services in Abbotsford

Several existing routes are being realigned and one route is being discontinued

Abbotsford council declines to declare climate emergency

Mayor says city is working to reduce emissions, but that procedures preclude immediate action

VIDEO: Unofficial goat yoga world record set in Abbotsford

Maan Farms waiting on verification from Guinness on world record

Vehicle roll-over causes Highway 1 delays

Left eastbound and westbound lanes blocked as crews deal with accident

Mt. Lehman community holds Canada Day parade

Event precedes main City of Abbotsford celebrations

VIDEO: Killer whale steals fisherman’s catch off North Coast

Fishing duel results in eager orca snagging salmon in Prince Rupert

Fate of accused in Canadian couple’s 1987 killings in jury’s hands

William Talbott’s lawyer says DNA doesn’t prove murder

Child killed after being hit in driveway on Vancouver Island

The driver of the vehicle remained at the crash scene and is fully cooperating

Eating sandwiches, putting on makeup behind the wheel could land you a fine

RCMP say if you cause an accident while eating you could be penalized

Cat badly hurt in animal trap was likely stuck for days, B.C. owner says

Blu, a three-year-old house cat, suffered severe damage to his hind leg after being stuck in trap for days

40 cats surrendered in apparent hoarding at B.C. home

Officers found the cats living among piles of garbage and feces, suffering from fleas

Vancouver Aquarium drops cetacean ban lawsuit in new lease agreement

Ocean Wise CEO Lasse Gustavsson called the updated lease an exciting new chapter for the aquarium

Plane veers off runway, into ditch at Langley Airport

Fire, ambulance, and police are on scene

Okanagan RCMP bike patrol rolls up on alleged stolen vehicle from Burnaby

The driver, a 30-year-old Kelowna man, has been held in custody and is facing possible charges of possession of stolen property and obstructing a police officer

Most Read