Kevin Collins: I recommend pairing this excellent Pew study with this paper about diversity among opt in online sample providers we were discussing last week, as I think they illustrate some important trade offs in research related to why people answer surveys osf.io/preprints/ps... 1/

I recommend pairing this excellent Pew study with this paper about diversity among opt in online sample providers we were discussing last week, as I think they illustrate some important trade offs in research related to why people answer surveys osf.io/preprints/ps... 1/

We just released a new analysis about the potential for opt-in surveys -- those where people volunteer to take surveys -- to provide misleading results, especially for young adults and Hispanics pewrsr.ch/49BMhqF

Online opt-in polls can produce misleading results, especially for young people and Hispanic adultspewrsr.ch We examine how an opt-in poll may have unintentionally misled the public about the sensitive issue of Holocaust denial among young Americans.

People may bear the cost in time and attention to answer surveys for either intrinsic motivations (e.g. interest) and/or extrinsic motivations (e.g. money), which of course are not mutually exclusive. But the extent to which different approaches lean on one or the other can affect resulting data 2/

The extrinsic motivations encourage fast and inattentive responses to get the payout at as little personal cost as possible, leading to problems with opt-ins like the one documented by Pew in this new study. 3/

This may be particularly problematic with micropayments model common in the marketplace paradigm such that people have to take a lot or studies to get a payout, but I don’t know of direct evidence on that. 4/

Conversely, relying on intrinsic motivations may not have the same data quality problem but has a distributional problem, in that interest is not uniform across the population producing non-response biases 5/

Personally I’m am more wary of the extrinsic motivation than the intrinsic ones, as strategies for handling non-response bias are well developed and removal on the basis of failing attention checks (attention also being non-uniformly distributed) may cause more problems than it solves 6/

But the best answer to me seems to be to reduce the cost of taking the survey in the first place so that less interest or monetary incentive is required to balance out that cost in favor of participation, by making surveys shorter, easy to understand, and not overly cognitively taxing 7/7

Post