Google's ad raters complain about workload and insecurity
Google has thousands of ad raters, mostly working from home for $15 per hour, assessing content on YouTube and other sites. Although the rating programme began in 2004 it gained in importance after more than 200 brands, including Walmart, PepsiCo and Verizon, suspended their advertising after ads were served next to extremist material. The raters are asked to make assessments of content that will help train Google's artificial intelligence algorithms to distinguish between acceptable and unacceptable content. However, anonymous former raters said their heavy workload coupled with the graphic nature of some content and a lack of clear communication from Google mean they can feel overworked.
Every day across the nation, people doing work for Google log on to their computers and start watching YouTube. They look for violence in videos. They seek out hateful language in video titles. They decide whether to classify clips as “offensive” or “sensitive.” They are Google’s so-called “ads quality raters,” temporary workers hired by outside agencies to render judgments machines still can't make all on their own. And right now, Google appears to need these humans' help—urgently.

YouTube, the Google-owned video giant, sells ads that accompany millions of the site’s videos each day. Automated systems determine where those ads appear, and advertisers often don’t know which specific videos their ads will show up next to. Recently, that uncertainty has turned into a big problem for Google. The company has come under scrutiny after multiple reports revealed that it had allowed ads to run against YouTube videos promoting hate and terrorism. Advertisers such as Walmart, PepsiCo, and Verizon ditched the platform and much of the wider Google ad network.

Google has scrambled to control the narrative, saying the media has overstated the problem of ads showing up adjacent to offensive videos. Flagged videos received “less than 1/1000th of a percent of the advertisers’ total impressions,” the company says. Google’s chief business officer, Philipp Schindler, says the issue affected a “very, very, very small” number of videos. But ad raters say the company is marshaling them as a force to keep the problem from getting worse.

'We know very well that human eyes—and human brains—need to put some deliberate thought into evaluating content.' Former Ad Rater

Because Google derives 90 percent of its revenue from advertisers, it needs to keep more from fleeing by targeting offensive content—fast. But users upload nearly 600,000 hours of new video to YouTube daily; it would take a small city of humans working around the clock to watch it all. That’s why the tech giant has emphasized that it’s hard at work developing artificially intelligent content filters, software that can flag offensive videos at a greater clip than ever before. "The problem cannot be solved by humans and it shouldn't be solved by humans," Schindler recently told Bloomberg.

The problem is, the company still needs humans to train that AI. So Google still depends on a phalanx of human workers to identify and flag offensive material to build the trove of data its AI will learn from. But eight current and former raters tell WIRED that, at a time when the company is increasingly reliant on ad raters' work, poor communication with Google and a lack of job stability are impairing their ability to do their jobs well.

“I’m not saying this is the entire reason for the current crisis,” says a former Google ad rater, who was not authorized to speak with WIRED about the program. “But I do believe the instability in the program is a factor. We raters train AI, but we know very well that human eyes—and human brains—need to put some deliberate thought into evaluating content.”

Human Tasks

Tech companies have long employed content moderators; as people upload and share more and more content, this work has become increasingly important to these internet giants. The ad raters WIRED spoke with explained that their role goes beyond monitoring videos. They read comment sections to flag abusive banter between users. They check all kinds of websites served by Google’s ad network to ensure they meet the company’s standards of quality. They classify sites by category, such as retail or news, and click links in ads to see if they work. And, as their name suggests, they rate the quality of ads themselves.

In March, however, in the wake of advertiser boycotts, Google asked raters to set that other work aside in favor of a "high-priority rating project” that would consume their workloads “for the foreseeable future,” according to an email the company sent them. This new project meant focusing almost exclusively on YouTube—checking the content of videos or entire channels against a list of things that advertisers find objectionable. “It’s been a huge change,” says one ad rater.

'I'm worried if I take too long on too many videos in a row I'll get fired.' Ad Rater

Raters say their workload suggests that volume and speed are more of a priority than accuracy. In some cases, they’re asked to review hours-long videos in less than two minutes. On anonymous online forums, raters swap time-saving techniques—for instance, looking up rap video lyrics to scan quickly for profanity, or skipping through a clip in 10-second chunks instead of watching the entire thing. A timer keeps track of how long they spend on each video, and while it is only a suggested deadline, raters say it adds a layer of pressure. "I'm worried if I take too long on too many videos in a row I'll get fired,” one rater tells WIRED.

Ad raters don’t just flag videos as inappropriate. They are asked to make granular assessments of their title and contents—classifying them, for instance, as containing “Inappropriate Language,” such as “profanity,” “hate speech,” or “other.” Or “Violence,” with the subcategories “terrorism,” “war and conflict,” “death and tragedy,” or “other.” There’s also “Drugs” and “Sex/Nudity” (with the subcategories “abusive,” “nudity,” or “other”). The system also provides the ad rater with an option for "other sensitive content"—if, say, someone is sharing extreme political views. (AdAge recently reported that Google is now allowing clients to opt out from advertising alongside “sexually suggestive” and “sensational and shocking” content, as well as content containing “profanity and rough language.”)

Some material doesn't always fit neatly into the provided categories, ad raters say. In those cases, raters label the material as "unrateable." One current rater described how he had to evaluate two Spanish-speaking people engaged in a rap battle. "I checked it as unrateable because of the foreign language," he told WIRED. "I also added a comment that said it seems like this is a video of people insulting each other in a foreign language, but I can't exactly tell if they are using profanity." (Judging from recent ad-rating job openings, one former rater said, Google seems to be prioritizing hiring bilingual raters. Workers can also check a box when a video is in a language they don’t understand.)

Multiple ad raters say they have been asked to watch videos with shocking content. “The graphic stuff is far more graphic lately … someone trying to commit suicide with their dog in their truck,” one rater said. The person set the truck on fire, the rater said, then exited the truck and committed suicide with a shot to the head. In the online forums frequented by ad raters, anonymous posters said they had seen videos of violence against women, children, and animals. Several posters said they needed to take breaks after watching several such videos in a row. Ad raters said they don’t know how Google selects the videos they will watch—they only see the title and thumbnail of the video before they rate it, not a rationale. Other typical content in videos raters are tasked to watch include people talking videogames, politics, and conspiracy theories.

Taken together, the scope of the work and nuance required in assessing videos shows Google still needs human help in dealing with YouTube’s ad problems. “We have many sources of information, but one of our most important sources is people like you,” Google tells raters in a document describing the purpose of their ad-rating work. But while only machine intelligence can grapple with YouTube’s scale, as company execs and representatives have stressed again and again, until Google’s machines—or anyone else’s—get smart enough to distinguish, say, truly offensive speech from other forms of expression on its own, such efforts will still need to rely on people.