When AI needs a human assistant

For years, Amazon’s Mechanical Turk (mTurk) has been a kind of open secret in the tech world, a place where fledgling algorithms can hire human labor on the cheap. If you need a hundred people to trace the boundaries of an object or fill out a survey, it’s the single best place to make it happen.

But while the project itself is well-known, it’s always slightly embarrassing when a company turns up there. In 2017, Expensify was spotted asking mTurk workers to enter data from receipts, leading the company to rush out a statement insisting that the mTurk project had nothing to do with Expensify’s main app. In part, it was a privacy issue, but mostly it was embarrassing: Expensify was built on a simple piece of technology — the ability to extract data from a photo of a receipt — and the mTurk tasks made it look like that technology was a sham. What if it was human beings extracting that data all along?

A turn through Mechanical Turk’s current listings shows those tactics are still alive and well. The biggest name in the current listings is Pinterest, which is currently offering Turkers 40 cents a pop to rule on whether a given post contains health misinformation. Pinterest has struggled with pseudoscience on its platform, with anti-vaccination posts becoming enough of a problem that all vaccine-related content has been banned on the platform since December. But while it’s easy to tell whether a post contains the word “vaccine,” it’s much harder to tell whether it’s making unrealistic health claims or pushing non-medical treatments for serious diseases. Guidelines attached to the task instruct the moderators to strike down exaggerated claims like “Kill sinus infection in 20 seconds,” while leaving up claims like “2 week flat belly challenge” because, as the guidelines put it, “an exercise challenge is not a[n] easy or quick fix.”

It’s hard to say how Mechanical Turk fits into Pinterest’s overall moderation system. The company did not respond to questions, but it’s likely whatever data is taken from Mechanical Turk is tempered with more reliable data from user-reporting and trained contractors, along with a trained algorithm to generate flags. Of course, each of those systems has their weaknesses (not least the horrible labor conditions), and sometimes, what you really need to train an algorithm is just an enormous volume of data. In those cases, mTurk is often the best place to get it. But it’s always disappointing to look under the hood of such a complex system, expecting to see an impeccably trained algorithm, only to find a bunch of poorly paid humans doing the dirty work.