The more I think about the NYT/Siena decision to include partial respondents, the more silly it seems.
For those who aren't familiar: in the NYT/Siena poll, people who hang up partway through the call are now included in the final results.
Under traditional polling methodologies, this wouldn't be possible, because in order for someone to be included in the poll you have to be able to weight their response by their demographics, and their demographics are unknown if they never get to the point of answering the demographic questions.
NYT/Siena gets around this by employing a seemingly clever two-stage weighting scheme. First, they weight the full respondents according to their normal demographic buckets. Then they take the partial respondents and weight *them* by the top line number.
In other words, if after weighting, the full respondents are Trump +1, they weight the partial respondents to match that Trump +1.
But wait. If you're weighting the partial respondents to match the top line, how can the top line be impacted by the partial respondents? What's even the point?
I believe the answer is trimming. In general, weights in surveys are trimmed to somewhat limit their impact. Yes, maybe you really expect black people with no college degree to be X% of the electorate, but if you get a weird sample and you actually target X%, it could result in huge swings
This is to say: weights introduce variance, and untrimmed weights can increase variance *a lot* (especially with small buckets). Trimming reduces that variance at the cost of introducing bias.
Which means... the *only* way the partial respondents can impact the top line number is, almost definitionally, through *bias*. If all of the bias were weighted away, they would have no impact.
Doesn't seem like a great methodology to me. Appreciate the effort, I guess, but non-response bias really is the Achilles heel of polling, and any methodology that doesn't somehow *actually* get *demographically identifiable* respondents to answer their phones seems pretty doomed.
For the record, I believe the top line impact is ~1 point or so. (The weighting guarantees that the effect of including these people will be muted.) But it's 1 point of statistical bias.
Too clever by half.
