  Semantic relatedness of terms represents similarity of meaning by a numerical
score. On the one hand, humans easily make judgments about semantic
relatedness. On the other hand, this kind of information is useful in language
processing systems. While semantic relatedness has been extensively studied for
English using numerous language resources, such as associative norms, human
judgments, and datasets generated from lexical databases, no evaluation
resources of this kind have been available for Russian to date. Our
contribution addresses this problem. We present five language resources of
different scale and purpose for Russian semantic relatedness, each being a list
of triples (word_i, word_j, relatedness_ij). Four of them are designed for
evaluation of systems for computing semantic relatedness, complementing each
other in terms of the semantic relation type they represent. These benchmarks
were used to organize a shared task on Russian semantic relatedness, which
attracted 19 teams. We use one of the best approaches identified in this
competition to generate the fifth high-coverage resource, the first open
distributional thesaurus of Russian. Multiple evaluations of this thesaurus,
including a large-scale crowdsourcing study involving native speakers, indicate
its high accuracy.

Computation and Language