  In this paper, we propose a novel information criteria-based approach to
select the dimensionality of the word2vec Skip-gram (SG). From the perspective
of the probability theory, SG is considered as an implicit probability
distribution estimation under the assumption that there exists a true
contextual distribution among words. Therefore, we apply information criteria
with the aim of selecting the best dimensionality so that the corresponding
model can be as close as possible to the true distribution. We examine the
following information criteria for the dimensionality selection problem: the
Akaike Information Criterion, Bayesian Information Criterion, and Sequential
Normalized Maximum Likelihood (SNML) criterion. SNML is the total codelength
required for the sequential encoding of a data sequence on the basis of the
minimum description length. The proposed approach is applied to both the
original SG model and the SG Negative Sampling model to clarify the idea of
using information criteria. Additionally, as the original SNML suffers from
computational disadvantages, we introduce novel heuristics for its efficient
computation. Moreover, we empirically demonstrate that SNML outperforms both
BIC and AIC. In comparison with other evaluation methods for word embedding,
the dimensionality selected by SNML is significantly closer to the optimal
dimensionality obtained by word analogy or word similarity tasks.

Machine Learning;Computation and Language;Machine Learning