Me, not a stats guy, but also not stats illiterate: “surely these models have an adequate mechanism for dealing with something as obvious as ‘old republicans are picking up the phone, while others aren’t, really.”
The Nates: “nope!”
It's always been CW that response rates may be responsible for temporary polling bumps. That they are responsible for longer term polling results, pollsters may not want to admit.
That’s something the individual pollsters try to do before putting out the poll results, by reducing the weight of responses from groups more likely to answer. But that can only do so much, if the response issue is so bad that there’s no good data to weight.
It’s part of why it’s silly to get too granular about the Nates’ approach of combining all the polls into a meta model. They’re combining weighted results of different polls with different methodologies and weights. Nice for a quick overview if you don’t want to dig into methodologies, that’s all.
I'm not a polling or stats expert, but my understanding is that the way they would typically correct for a response bias is by projecting the results using models of prior electorates.
The real problems is that we're *also* experiencing a major political realignment
So, like, the prior electorates don't map cleanly onto the current electorates. So, one of the main strategies for resolving these kinds of response rate issues is severely impacted by the *other* major trend affecting pollsters
With how spammy phones have gotten, I’m afraid we’ve reached the point where “people who answer unknown number” are not representative of the the voting population
Totally. My kid, who will vote, looked at me like I had two heads when I suggested they answer the phone. “I never answer the phone when it’s an unknown number” they said. (I was like, suppose it’s a job or something else important? They said they check their phone messages.)
They correct for this by weighting by recalled Trump vote.
The topline results are close to correct, but for anything other than the presidential horserace you'll have too many hardcore Rs and not enough moderate Rs.
At some points in 2020, especially after the strictest lockdowns eased, polls had the opposite problem: Democrats would eagerly pick up the phone and getting Republicans was like pulling teeth. Proper sample weighting can help to an extent, but can’t entirely cure it.
so I guess the idea here for why this might skew the polls is that if you have two separate groups of republicans—high-motivated, praise-for-dear-leader group, and low-motivation, anti-trump group—you'll be talking to a lot more of the former than the latter when you poll
Translation: Democrats have families and hobbies and stuff to do. Republicans are eager to have *anyone* at which to air their petty grievances, and the pollsters aren't allowed to hang up on them (until they get REALLY racist).
Last election I got a pollster for the first time and I forced myself to answer questions because I kinda had a hunch this sort of thing was going on. I wanted to hang up so bad.
The increasing difficulties with getting a quality sample are one significant reason why polls have been a tad borked for awhile now.
I can see how eventually they lose most of their predictive value and will be only useful in spotting "movement".
