← Statistics and Research Methods › Data Collection

While Amazon's Mechanical Turk (MTurk) has reduced the cost of collecting original data, in 2018, researchers noted the potential existence of a large number of bad actors on the platform. To evaluate data quality on MTurk, we fielded three surveys between 2018 and 2020. While find no evidence of a "bot epidemic," significant portions of the data—between 25%-35%—are of dubious quality. While the number of IP addresses that completed the survey multiple times or circumvented location requirements fell almost 50% over time, suspicious IP addresses are more prevalent on MTurk than on other platforms. Furthermore, many respondents appear to respond humorously or insincerely, and this behavior increased over 200% from 2018-2020. Importantly, these low-quality responses attenuate observed treatment effects by magnitudes ranging from approximately 10-30%.

Cite: Gaurav Sood, Doug Ahler, and Carrie Roush (2025). "The Micro-Task Market for "Lemons": Collecting Data on Amazon's Mechanical Turk." Political Science Research and Methods, 13(1), 1–20, 2025. https://doi.org/10.1017/psrm.2021.57

BibTeX:
@article{sood2025the,
  author = {Gaurav Sood and Doug Ahler and Carrie Roush},
  title = {{The Micro-Task Market for "Lemons": Collecting Data on Amazon's Mechanical Turk}},
  journal = {Political Science Research and Methods},
  volume = {13},
  number = {1},
  pages = {1--20},
  year = {2025},
  doi = {10.1017/psrm.2021.57},
  url = {https://doi.org/10.1017/psrm.2021.57}
}