  The TREC Real-Time Summarization (RTS) track provides a framework for
evaluating systems monitoring the Twitter stream and pushing tweets to users
according to given profiles. It includes metrics, files, settings and
hypothesis provided by the organizers. In this work, we perform a thorough
analysis of each component of the framework used in 2016 and 2017 and found
some limitations for the Scenario A of this track. Our main findings point out
the weakness of the metrics and give clear recommendations to fairly reuse the
collection.

Information Retrieval