Reuters uses AI to mine Twitter for stories
News agency Reuters is using artificial intelligence (AI) to automatically identify breaking news stories. The company's research and development team has created a system called Reuters Tracer, which uses Twitter to monitor news events as they happen. The system automatically selects the most relevant and writes a headline and summary, which are then sent around the company's global distribution channels. The software analyses around 12 million tweets a day, half of which come from accounts curated by Reuters journalists, and half of which are randomly sampled. The company claims the system has worked well in tests, generating accurate alerts quickly.

Nevertheless, news agencies such as the Associated Press have moved ahead with automated news writing services. These report standard announcements such as financial news and certain sports results by pasting the data into pre-written templates: “X reported profit of Y million in Q3, in results that beat Wall Street forecasts ... ”

So there is significant pressure on other news agencies to automate news production. And today, Reuters outlines how it has almost entirely automated the identification of breaking news stories. Xiaomo Liu and pals at Reuters Research and Development and Alibaba say the new system performs well. Indeed, it has the potential to revolutionize the news business. But it also raises concerns about how such a system could be gamed by malicious actors.

The new system is called Reuters Tracer. It uses Twitter as a kind of global sensor that records news events as they are happening. The system then uses various kinds of data mining and machine learning to pick out the most relevant events, determine their topic, rank their priority, and write a headline and a summary. The news is then distributed around the company’s global news wire.

The first step in the process is to siphon the Twitter data stream. Tracer examines about 12 million tweets a day, 2 percent of the total. Half of these are sampled at random; the other half come from a list of Twitter accounts curated by Reuters’s human journalists. They include the accounts of other news organizations, significant companies, influential individuals, and so on.

The next stage is to determine when a news event has occurred. Tracer does this by assuming that an event has occurred if several people start talking about it at once. So it uses a clustering algorithm to find these conversations.

Of course, these clusters include spam, advertisements, ordinary chat, and so on. Only some of them refer to newsworthy events.

So the next stage is to classify and prioritize the events. Tracer uses a number of algorithms to do this. The first identifies the topic of the conversation. It then compares this with a database of topics that the Reuters team has gathered from tweets produced by 31 official news accounts, such as @CNN, @BBCBreaking, and @nytimes as well as news aggregators like @BreakingNews.

At this stage, the algorithm also determines the location of the event using a database of cities and location-based keywords.