Pollsters still searching for fixes as 2022 voting nears

If ABC and The Washington Post were to be believed in the days before Election Day 2020, then-candidate Joseph R. Biden was about to wipe the floor with President Trump in Wisconsin. The news outlets’ Oct. 28 survey, just a week before voting day, showed the Democrat up by a staggering 17 percentage points.

Mr. Biden would win the state, though only by the barest of margins: 63 hundredths of a percentage point. ABC and The Post weren’t just off base; they were in a different galaxy where Republican voters didn’t seem to exist.

Missing Republicans plagued pollsters across the board in surveys ahead of the 2020 election, leading to bungled projections and a new round of soul-searching by the industry.

Yet as voters prepare for November’s midterm elections, it’s not clear whether the pollsters have solved anything.

“In the media polls, no, they have not fixed it,” said Jim McLaughlin, a Republican Party pollster. “I used to think it was incompetence back in 2016. Now I think they do it purposely to tamp down on Republican support.”

Polls are getting plenty of attention these days, particularly as they show Republicans suddenly struggling in what should be a good environment for them: a midterm election year with a historically unpopular Democratic president, a lingering pandemic and a chaotic economy.

The fundamental trick to polling is finding a sample that matches the overall population well enough to get valid results. In decades past, when nearly everybody who voted had a landline telephone and wasn’t bombarded by scam calls, it was relatively easy to get a solid sample.

In 1997, Gallup says, it got responses from 28% of those it dialed. By 2017, that had dropped to just 7%. Pollsters have gone looking for ways to supplement — or in some cases completely supplant — phone calls.

Election polling, or “horse race” surveys, have another wrinkle: They never know who, exactly, will show up to cast a ballot.

Many pollsters compensate by trying to identify “likely” voters and “weight” their results to match what they believe will be the turnout.

A survey of 1,000 people in a generally red-leaning state that ended up with 500 self-identified Democrats in the sample, for example, would be weighted to more closely match voter registration numbers or an expected partisan turnout.

Michael McKenna, a top legislative aide in the Trump White House who also conducts polling, said pollsters spend far too much time trying to guess the turnout factor and end up introducing major errors to their work.

“The thing about turnout models is, they’re great, right? They can be predictive if you’re right. But if you’re wrong about who shows up, guess what? The whole thing is wrong,” said Mr. McKenna, who writes a column for The Washington Times.

He said readers should be wary of polls that tout weighting.

“These guys are afraid of their own methodology, which is why they weight. Once you get something that’s weighted, you’ve essentially substituted your own opinion for science,” Mr. McKenna said.

He said some polls feel so wrong that they should never be released. Yet media organizations that have pumped tens of thousands of dollars into surveys feel compelled to run their work, and outlandish results can perversely get more clicks online.

The ABC/Post 2020 Wisconsin poll showing Mr. Biden winning by 17 points is a telling example.

That survey was based on a partisan breakdown of 30% Democrats, 26% Republicans and 44% something else. By contrast, the 2020 electorate was 32% Democrats, 37% Republicans and 31% something else.

The poll underestimated the share of Republicans by 11 percentage points.

Mr. McLaughlin said the Wisconsin poll was so stark that he got a call from Mr. Trump asking about it.

In a statement to The Washington Times, The Post said that poll was “well off the mark” and pointed out that it alerted readers to that fact at the time. In the eighth paragraph of its report, the paper said its sample showed slightly more Clinton voters than Trump voters. If the survey had been weighted to match the 2016 electorate, Mr. Biden’s lead would have been cut to 12 points.

The Post said it and ABC have taken steps to improve polling quality, including sharing raw data sets from their 2020 polls and inviting independent analysis.

In 2021, when The Post polled in the Virginia governor’s race, the paper published an article on how it conducted and adjusted its polling to correct for errors from 2020. The paper’s final poll showed Democrat Terry McAuliffe up 1 percentage point over Republican Glenn Youngkin, which was within the margin of error. Mr. Youngkin won by about 2 percentage points.

A Post employee was also part of the American Association for Public Opinion Research’s post-2020 task force, which tried to figure out what went wrong.

Among the potential problems they identified were oversampling of Democratic voters, wrongly correcting for errors from the 2016 vote and failing to spot millions of new voters who turned out.

Some analysts argue that pollsters and the poll-consuming public need a refresher.

G. Elliott Morris says in a new book, “Strength in Numbers: How Polls Work and Why We Need Them,” that journalists need to remind audiences of all the ways a poll might be off and to stop treating a survey showing a lead within the margin of error as a clear indication that one candidate is winning. That’s a toss-up race, Mr. Morris says.

Many poll consumers have developed shortcuts for interpretation, such as assuming a standard level of bias. Some readers automatically adjust the results of any ABC/Washington Post survey a few percentage points more toward Republicans.

John Cluverius, associate director of the Center for Public Opinion at the University of Massachusetts, Lowell, cautioned against that approach, particularly for midterm elections where contests are state-by-state. There is no single consistent formula that works everywhere all the time, he said.

He pointed to work his outfit did in 2020 in North Carolina, New Hampshire and Texas. Its polling was accurate in North Carolina and reasonably close in New Hampshire, but it missed badly in Texas, he said. Other pollsters did better in Texas but missed in North Carolina.

“What regular consumers of news and regular consumers of polling information should know is most pollsters out there are doing the best they can with the resources they have. If they are being open and transparent about their methodology, about what went right and what went wrong, that is the most we could ask for.”

He said polling in the current political environment is particularly tricky because unusually large groups of people are experiencing divergent economies.

“We’re trying to apply all these textbook rules to a situation that isn’t described in any textbook,” he said.

Mr. McLaughlin said good rules of thumb for pollsters are to use voter lists rather than random samplings of adults. His firm also uses a wide range of methods to reach potential respondents, including cellphones, a small percentage of landlines, and text messages.

“It costs us like thousands of dollars just to build the list to make sure we’ve got a good sample,” he said. “Good polling is expensive.”

That’s one reason why those in the business speak differently about polling for penny-pinching media organizations or polling meant for public consumption.

“Its use is almost expressly for propaganda. It’s for the press, it’s for donors, it’s to show people ‘Hey, we’re winning’ or ‘Hey, we’re gaining,’” Mr. McKenna said.

“I believe that we’re going to see a lot of the same mistakes replicated in 2022,” he said. “The midterms are harder to do survey work just because you’re not quite sure who’s going to show up. And in this environment, guys with turnout models and weighted averages are inclined to use them more generously.”