Usually bilingual word vectors are trained “online”. Mikolov et al. (2013a) showed they can also be found “offline”; whereby two pre-trained embeddings are aligned with a linear transformation, using dictionaries compiled from expert knowledge. In this work, we prove that the linear transformation between two spaces should be orthogonal. This transformation can be obtained using the singular value decomposition. We introduce a novel “inverted softmax” for identifying translation pairs, with which we improve the precision @1 of Mikolov’s original mapping from 34% to 43%, when translating a test set composed of both common and rare English words into Italian. Orthogonal transformations are more robust to noise, enabling us to learn the transformation without expert bilingual signal by constructing a “pseudo-dictionary” from the identical character strings which appear in both languages, achieving 40% precision on the same test set. Finally, we extend our method to retrieve the true translations of English sentences from a corpus of 200k Italian sentences with a precision @1 of 68%.
1 INTRODUCTION
Monolingual word vectors embed language in a high-dimensional vector space, such that the similarity of two words is defined by their proximity in this space (Mikolov et al., 2013b). They enable us to train sophisticated classifiers to interpret free flowing text (Kim, 2014), but they require independent models to be trained for each language. Crucially, training text obtained in one language cannot improve the performance of classifiers trained in another, unless the text is explicitly translated. Increasing interest is now focused on bilingual vectors, in which words are aligned by their meaning, irrespective of the language of origin. Such vectors may drive improvements in machine translation (Zou et al., 2013), and enable language-agnostic text classifiers (Klementiev et al., 2012). They can also be higher quality than monolingual vectors (Faruqui & Dyer, 2014).
Bilingual vectors are normally trained “online”, whereby both languages are learnt together in a shared space (Chandar et al., 2014; Hermann & Blunsom, 2013). Typically these algorithms exploit two sources of monolingual text alongside a smaller bilingual corpus of aligned sentences. This bilingual signal provides a regularisation term, which penalises the embeddings if similar words in the two languages do not lie nearby in the vector space. However Mikolov et al. (2013a) showed that bilingual word vectors can also be obtained “offline”. Two sets of word vectors in different languages were first obtained independently, and then a linear matrix W was trained using a dictionary to map word vectors from the “source” language into the “target” language. Remarkably, this simple procedure was able to translate a test set of English words into Spanish with 33% precision.
To develop an intuition for these two approaches, we note that the similarity of two word vectors is defined by their cosine similarity, cos(θij) = yTi xj/|yi||xj |. The vectors have no intrinsic meaning, it is only the angles between vectors which are meaningful. This is closely analogous to asking a cartographer to draw a map of England with no compass. The map will be correct, but she does not know which direction is north, so the angle of rotation will be random. Two maps drawn by two such cartographers will be identical, except that one will be rotated by an unknown angle with respect to the other. There are two ways the cartographers could align their maps. They could draw
the maps together, thus ensuring that landmarks are placed nearby on both maps during “training”. Or they could draw their maps independently, and then compare the two afterwards; rotating one map with respect to the other until the major cities are aligned. We note that the more similar the intrinsic geometry of the two maps, the more accurately this rotation will align the space.
The main contribution of this work is to provide theoretical insights which unify and enhance existing approaches in the literature. We prove that a self-consistent linear transformation between vector spaces should be orthogonal. Intuitively, the transformation is a rotation, and it is found using the singular value decomposition (SVD). The shared semantic space obtained by the SVD is illustrated in figure 1. We build on the work of Dinu et al. (2014), by introducing a novel “inverted softmax” to combat the hubness problem. Using the same word vectors, training dictionary and test set provided by Dinu, we improve the precision of Mikolov’s method from 34% to 43%, when translating from English to Italian, and from 25% to 37% when translating from Italian to English. We also present three remarkable new results. First we exploit the superior robustness of orthogonal transformations, by discarding the training dictionary and forming a pseudo-dictionary from the identical character strings which appear in both languages. While Mikolov’s method achieves translation precisions of just 1% and 3% respectively with this pseudo-dictionary, our approach achieves precisions of 40% and 34%. This is a striking result, achieved without any expert bilingual signal. Next, we form simple sentence vectors by summing and normalising over word vectors, and we obtain bilingual sentence vectors by applying the SVD to a phrase dictionary formed from a bilingual corpus of aligned text. The transformation obtained aligns the underlying word vectors, achieving a translation precision of 43% and 38%, en-par with the expert word dictionary above. Finally, we show that we can also use our bilingual word vectors to retrieve sentence translations; identifying the translation of an English sentence from a bag of 200k Italian candidate sentences with 68% precision.
2 OFFLINE BILINGUAL LANGUAGE VECTORS
2.1 PREVIOUS WORK
Offline bilingual word vectors were first proposed by Mikolov et al. (2013a). They obtained a small dictionary of paired words from Google Translate, whose word vectors we denote {yi, xi}ni=1. Next, they applied a linear transformation W to the source language and used stochastic gradient descent to minimise the squared reconstruction error,
min W n∑ i=1 ||yi −Wxi||2. (1)
After training, any word vector in the source language can be mapped to the target by calculating ye = Wx. The similarity between a source vector x and a target vector yt can then be evaluated by the cosine similarity cos(θte) = yTt ye/|yt||ye|. Astonishingly, this simple procedure achieved 33% accuracy when translating unseen words from English into Spanish, using a training dictionary of
5k common English words and their Spanish translations, and word vectors trained using word2vec on the WMT11 text datasets. Translations were found by a simple nearest neighbour procedure.
We note that the cost function above is solved by the method of least squares, as realised by Dinu et al. (2014). They did not modify this cost function, but proposed an adapted method of retrieving translation pairs which was more accurate when translating words from English to Italian. Faruqui & Dyer (2014) obtained bilingual word vectors using CCA. They did not attempt any translation tasks, but showed that the combination of CCA and dimensionality reduction improved the performance of monolingual vectors on standard evaluation tasks. CCA had previously been used to iteratively extract translation pairs directly from monolingual corpora (Haghighi et al., 2008). More recently, Xing et al. (2015) argued that Mikolov’s linear matrix should be orthogonal, and introduced an approximate procedure composed of gradient descent updates and repeated applications of the SVD. CCA has been extended to map 59 languages into a single shared space (Ammar et al., 2016), and non-linear “deep CCA” has been introduced (Lu et al., 2015). A theoretical analysis of bilingual word vectors similar to this paper was recently published by Artetxe et al. (2016).
2.2 THE SIMILARITY MATRIX AND THE ORTHOGONAL TRANSFORM
To prove that a self-consistent linear mapping between semantic spaces must be orthogonal, we form the similarity matrix, S = YWXT . X and Y are word vector matrices for each language, in which each row contains a single word vector, denoted by lower case x and y. The matrix element,
Sij = y T i Wxj (2)
= yi · (Wxj), (3) evaluates the similarity between the jth source word and the ith target word. The matrix W maps the source language into the target language. The largest value in a column of the similarity matrix gives the most similar target word to a particular source word, while the largest value in a row gives the most similar source word to a given target word. However we could also form a second similarity matrix S′ = XQY T , such that the matrix Q maps the target language back into the source.
S′ji = x T j Qyi (4)
= xj · (Qyi), (5) also evaluates the similarity between the jth source word and the ith target word. To be self consistent, we require S′ = ST . However ST = XWTY T , and therefore the matrix Q = WT . If W maps the source language into the target, then WT maps the target language back into the source.
When we map a source word into the target language, we should be able to map it back into the source language and obtain the original vector. x ∼WT y and y ∼Wx and thus x ∼WTWx. This expression should hold for any word vector x and thus we conclude that the transformationW should be an orthogonal matrix O satisfying OTO = I , where I denotes the identity matrix. Orthogonal transformations preserve vector norms, so if we normalise X and Y , then the matrix element Sij = |yi||Oxj | cos(θij) = cos(θij). The similarity matrix S = Y OXT computes the cosine similarity between all possible pairs of source and target words under the orthogonal transformation O.
We now infer the orthogonal transformation O from a dictionary {yi, xi}ni=1 of paired words. Since we predict the similarity of two vectors by evaluating Sij = cos(θij), we ought to learn the transformation by maximising the cosine similarity of translation pairs in the dictionary,
max O n∑ i=1 yTi Oxi, subject to O TO = I. (6)
The solution proceeds as follows. We form two ordered matrices XD and YD from the dictionary, such that the ith row of {XD, YD} corresponds to the source and target language word vectors of the ith pair in the dictionary. We then compute the SVD of M = Y TDXD = UΣV
T . This step is highly efficient, since M is a square matrix with the same dimensionality as the word vectors. U and V are composed of columns of orthonormal vectors, while Σ is a diagonal matrix containing the singular values. Our cost function is minimised by O = UV T . The optimised similarity matrix,
S = Y UV TXT . (7) Sij = y T i UV Txj (8)
= (UT yi) · (V Txj). (9)
Thus, we map both languages into a single space, by applying the transformation V T to the source language and UT to the target language. We prove that this procedure maximises equation 6 in the appendix. It was recently independently proposed by Artetxe et al. (2016), and provides a numerically exact solution to the cost function proposed by Xing et al. (2015), just as the method of least squares provides a numerically exact solution to the cost function of Mikolov et al. (2013a).
Our procedure did not use the singular values Σ, but these values do carry relevant information. All of the singular values are positive, and each singular value si is uniquely associated to a pair of normalised vectors ui and vi from the matrices U and V . Standard implementations of the SVD return the singular values in descending order. The larger the singular value, the more rapidly the mean cosine similarity of the dictionary decreases if the corresponding vectors are distorted. We can perform dimensionality reduction by neglecting the vectors {ui, vi} which arise from the smallest singular values. This is trivial to implement by simply dropping the final few rows of UT and V T , and we will show below that it leads to a small improvement in the translation performance.
2.3 THE SVD AND CCA
Our method is very similar to the CCA procedure proposed by Faruqui & Dyer (2014), which can also be obtained using the SVD (Press, 2011). We first obtain the source dictionary matrix XD and subtract the mean from each column of this matrix to obtain X ′D. We then perform our first SVD to obtainX ′D = QDΣXV T X . We perform the same two operations on the target dictionary YD to obtain Y ′D = WDΣY V T Y , and then perform another SVD on the product M ′ = QTDWD = U ′Σ′V ′T . This last step is identical to the alignment procedure we introduced above. Finally we obtainX ′ and Y ′ by subtracting the mean value of each column in XD and YD, before computing a new pair of aligned representations of the full vocabulary, Qaligned = X ′VXΣ−1X U ′, and Waligned = Y ′VY Σ−1Y V ′. Once again, we perform dimensionality reduction by neglecting the final few columns of U ′ and V ′.
Effectively, CCA is composed of two stages. In the first stage, we replace our word vector matrices X and Y by two new vector representations Q = X ′VXΣ−1X and W = Y ′VY Σ −1 Y . In the second stage, we apply the orthogonal transformations {U ′, V ′} to align {Q,W} in a single shared space. To the authors, the first stage appears redundant. If we have already learned high-quality word vectors {X,Y }, there seems little reason to learn new representations {Q,W}. Additionally, it is unclear why the transformations VXΣ−1X and VY Σ −1 Y are obtained using only the dictionary matrices {XD, YD}, rather than using the full vocabularies {X,Y }.
2.4 THE INVERTED SOFTMAX
Mikolov et al. (2013a) predicted the translation of a source word xj by finding the target word yi closest to Wxj . In our formalism, this corresponds to finding the largest entry in the jth column of the similarity matrix. To estimate our confidence in this prediction, we could form the softmax,
Pj→i = eβSij∑ m e βSmj . (10)
To learn the “inverse temperature” β, we maximise the log probability over the training dictionary,
max β ∑ pairs ij ln(Pj→i). (11)
This sum should be performed only over valid translation pairs. Dinu et al. (2014) demonstrated that nearest neighbour retrieval is flawed, since it suffers from the presence of “hubs”. Hubs are words which appear as the nearest neighbour target word to many different source words, reducing the translation performance. We propose that the hubness problem is mitigated by inverting the softmax, and normalising the probability over source words rather than target words.
Pj→i = eβSij αj ∑ n e βSin . (12)
Intuitively, rather than asking whether the source word translates to the candidate target word, we assess the probability that the candidate target word translates back into the source word. We then select the target word which maximises this probability. If the ith target word is a hub, then the
denominator in equation 12 will be large, preventing this target word from being selected. The vector α ensures normalisation. The sum over n should run over all source words in the vocabulary. However to reduce the computational cost, we only perform this sum over ns sample words, chosen randomly from the vocabulary. Unless explicitly stated, ns = 1500.
2.5 PSEUDO DICTIONARIES
2.5.1 IDENTICAL CHARACTER STRINGS
Our method requires a training dictionary of paired vectors, which is used to infer the orthogonal map O and the inverse temperature β, and also as a validation set during dimensionality reduction. Typically this dictionary is obtained by translating common source words into the target language using Google Translate, which was constructed using expert human knowledge. However most European languages share a large number of words composed of identical character strings. Words like “London”, “DNA” and “Tortilla”. It is probable that identical strings across two languages share similar meanings. We can extract these strings and form a “pseudo-dictionary”, compiled without any expert bilingual knowledge. Below we show that this pseudo dictionary is sufficient to successfully translate between English and Italian with high precision.
2.5.2 ALIGNED SENTENCES
The Europarl corpus is composed of aligned sentences in a number of European languages (Koehn, 2005). Chandar et al. (2014) showed that such corpora can be used alongside monolingual text sources to learn online bilingual vectors, but to date, offline bilingual vectors have only been obtained from dictionaries. To learn the orthogonal transformation from aligned sentences, we define the vector q of a source language sentence by a normalised sum over the word vectors present, q = ∑ i xi/ | ∑ i xi|. The vector w of a target language sentence is defined by a normalised sum of word vectors yi. We view the aligned corpus as a dictionary of paired sentences {wi, qi}, from which we form two dictionary matrices WD and QD. We obtain the transformation O from an SVD on the matrix M = WTDQD, and use this transformation to translate individual words in the test set.
This simple procedure embeds words and sentences in the same vector space. The sentence embedding can be thought of as the “average word” that the sentence conveys. Intuitively, each aligned sentence pair gives us weak information about a possible word pair in the dictionary. By combining a large number of such sentence pairs, we obtain sufficient information to align the vector spaces and infer the translations of individual words. However, we will go on to show that this orthogonal transformation can be used, not only to retrieve the translations of words between languages, but also to retrieve the translations of sentences between languages with remarkably high accuracy.
3 EXPERIMENTS
We perform our experiments using the same word vectors, training dictionary and test dictionary provided by Dinu et al. (2014)1. The word vectors were trained using word2vec, and then the 200k most common words in both the English and Italian corpora were extracted. The English word vectors were trained on the WackyPedia/ukWaC and BNC corpora, while the Italian word vectors were trained on the WackyPedia/itWaC corpus. The training dictionary comprises 5k common English words and their Italian translations, while the test set is composed of 1500 English words and their Italian translations. This test set is split into five sets of 300. The first 300 words arise from the most common 5k words in the English corpus, the next 300 from the 5k-20k most common words, followed by bins for the 20k-50k, 50k-100k, and 100k-200k most common words. This enables us to evaluate how word frequency affects the translation performance. Some of the Italian words have both male and female forms, and we follow Dinu in considering either form a valid translation.
We report results using our own procedure, as well as the methods proposed by Mikolov, Faruqui, and Dinu. We compute results for Mikolov’s method by applying the method of least squares, and results for Faruqui’s method using Scikit-learn’s implementation of CCA with default parameters. In both cases, we predict translations by nearest neighbour retrieval. We do not apply dimensionality
1These are available at http://clic.cimec.unitn.it/~georgiana.dinu/down/
reduction following CCA, to enable a fair comparison with our SVD procedure. We compute results for Dinu’s method using the source code they provided alongside their manuscript. Their method uses 10k source words as “pivots”; 5k from the test set and 5k chosen at random from the vocabulary. By contrast, the inverted softmax does not know which source words occur in the test set.
3.1 EXPERIMENTS USING THE EXPERT TRAINING DICTIONARY
In tables 1 and 2 we present the translation performance of our methods when translating the test set between English and Italian, using the expert training dictionary provided by Dinu. We evaluate Mikolov and Dinu’s methods for comparison, as well as CCA- proposed by Faruqui & Dyer (2014). All the methods are more accurate when translating from English to Italian. This is unsurprising given that some English words in the test set can translate to either the male or female form of the Italian word. In the fourth column we evaluate the performance of our SVD procedure with nearest neighbour retrieval. This already provides a marked improvement on Mikolov’s mapping, especially when translating from Italian into English. As anticipated, the performance of the SVD is very similar to CCA. In the following two columns we apply first the inverted softmax, and then dimensionality reduction to the aligned vector space obtained by the SVD. The hyper-parameters of these procedures were optimised on the training dictionary. Combining both procedures improves the precision @1 to 43% and 38% when translating from English to Italian, or Italian to English respectively. These results are a significant improvement on previous work. In table 3 we present the dependence of precision @1 on word frequency. We achieve remarkably high precision when translating common words. This performance drops off for rare words; presumably either because there is insufficient monolingual data to learn high quality rare word vectors, or because the linguistic similarities between rare words across languages are less pronounced.
3.2 EXPERIMENTS USING IDENTICAL CHARACTER STRINGS
In the preceding section, we reported our performance using an orthogonal transformation learned on an expert training dictionary of 5k common English and Italian words. We now report our performance when we do not use this dictionary, and instead construct a pseudo dictionary from the list of words which appear in both the English and Italian vocabularies, composed of exactly the same character string. Remarkably, 47074 such identical character strings appear in both vocabularies. There would be fewer identical entries for more diverse language pairs, but our main goal here is to demonstrate the superior robustness of orthogonal transformations to low quality dictionaries.
We exhibit our results in table 4, where we evaluate our method (SVD + inverted softmax + dimensionality reduction), when translating either from English to Italian or from Italian to English. Even when using this pseudo dictionary prepared with no expert bilingual knowledge, we still achieve a mean translation performance @1 of 40% from English to Italian on our test set. By contrast, Mikolov and Dinu’s methods achieve precisions of just 1% and 6% respectively. CCA also performs well, although it became significantly more computationally expensive when the vocabulary size increased. Previously translation pairs have been extracted from monolingual corpora using CCA by bootstrapping a small seed lexicon (Haghighi et al., 2008).
3.3 EXPERIMENTS ON THE EUROPARL CORPUS OF ALIGNED SENTENCES
The English-Italian Europarl corpus comprises 2 million English sentences and their Italian translations, taken from the proceedings of the European parliament (Koehn, 2005). As outlined earlier, we can form simple sentence vectors in the word vector space by summing and normalising over the words contained in a sentence. These sentence vectors can be used in two different tasks. First, we can use the Europarl corpus as a training dictionary, whereby the matrices XD and YD are formed from the sentence vectors of translation pairs. By applying the SVD to the first 500k sentences in this “phrase dictionary”, we obtain a set of bilingual word vectors from which we can retrieve translations of individual words. We exhibit the translation performance of this approach in table 5. We achieve 42.8% precision @1 when translating from English into Italian and 37.5% precision when translating from Italian into English, comparable to the accuracy achieved using the expert word dictionary on the same test set. It is difficult to compare the two approaches, since they require different training data. However our performance appears competitive with Bilbowa, a leading method for learning bilingual vectors online from monolingual corpora and aligned text (Gouws et al., 2015). We do not include results for CCA due to the computational complexity on a dictionary of this size.
Second, we can apply our orthogonal transformation to retrieve the Italian translation of an English sentence, or vice versa. To achieve this, we hold back the final 200k English and Italian sentences from our 500k sample of Europarl, and attempt to retrieve the true translation of a given sentence in this test set. We obtain the orthogonal transformation by performing the SVD on either the expert word dictionary provided by Dinu, or on a phrase dictionary formed from the first 300k
sentences from Europarl. For simplicity, we do not apply dimensionality reduction here. Our results are provided in table 6. For precision @1, most approaches favour the phrase dictionary, while Dinu’s method favours the word dictionary. We show in the appendix that all methods favour the phrase dictionary for precision @5 and @10. Remarkably, given no information except the sentence vectors, we are able to retrieve the correct translation of an English sentence with 67.8% precision. This is particularly surprising, since we are using the simplest possible sentence vectors, which have no information about word order or sentence length. It is likely that we could improve on these results if we used higher quality sentence vectors (Le & Mikolov, 2014; Kiros et al., 2015), although we might lose the ability to simultaneously align the underlying word vector space.
When training the inverted softmax, the inverse temperature β diverged, and the “translation” performance from English to Italian significantly exceeded the performance from Italian to English. This suggested that sentence retrieval from Italian to English might be achieved better by nearest neighbours, so we also evaluated the performance of nearest neighbour retrieval on the same orthogonal transformation, as shown in the third row of table 6. This improved the performance from Italian to English from 48.6% to 65.6%, which suggests that the optimal retrieval approach would be able to tune continuously between the conventional softmax and the inverted softmax.
4 SUMMARY
We proved that the optimal linear transformation between word vector spaces should be orthogonal, and can be obtained by a single application of the SVD on a dictionary of translation pairs, as proposed independently by Artetxe et al. (2016). We used the SVD to obtain bilingual word vectors, from which we can predict the translations of previously unseen words. We introduced a novel “inverted softmax” which significantly increased the accuracy of our predicted translations. Combining the SVD with the inverted softmax and dimensionality reduction, we improved the translation precision of Mikolov’s original linear mapping from 34% to 43%, when translating a test set composed of both common and rare English words into Italian. This was achieved using a training dictionary of 5k English words and their Italian translations. Replacing this training dictionary with a pseudodictionary acquired from the identical word strings that appear in both languages, we showed that we still achieved 40% precision, demonstrating that it is possible to obtain bilingual vector spaces without an expert bilingual signal. Mikolov’s method achieves just 1% precision here, emphasising the superior robustness of orthogonal transformations. There are currently a number of approaches to obtaining offline bilingual word vectors in the literature. Our work shows they can all be unified.
Finally, we defined simple sentence vectors to obtain offline bilingual word vectors without a dictionary using the Europarl corpus. We achieved 43% precision when translating our test set from English into Italian under this approach, comparable to our results above, and competitive with online approaches which use aligned text as the bilingual signal. We demonstrated that we can also use our sentence vectors to retrieve the true translation of an English sentence from a bag of 200k Italian candidate sentences with 68% precision, a striking result worthy of further investigation.
ACKNOWLEDGMENTS
We thank Dinu et al. for providing their source code, pre-trained word vectors, and a training and test dictionary of English and Italian words, and Philipp Koehn for compiling the Europarl corpus.
A THE ORTHOGONAL PROCRUSTES PROBLEM
There is not an intuitive analytic solution to the cost function in equation 6, but an analytic solution does exist to the closely related “orthogonal Procrustes problem”, which minimises the squared reconstruction error subject to an orthogonal constraint (Schönemann, 1966),
min O n∑ i=1 ||yi −Oxi||2, subject to OTO = I. (13)
However both X and Y are normalised, while O preserves the vector norm. We note that,
||yi −Oxi||2 = |yi|2 + |xi|2 − 2yTi Oxi (14) ∝ A− yTi Oxi. (15)
A is a constant, and so the cost functions given in equations 6 and 13 are equivalent. We presented the solution of the orthogonal Procrustes problem in the main text.
B ADDITIONAL EXPERIMENTS
In tables 7 and 8, we provide results at precisions @5 and @10, for the same experiments shown @1 in table 6 of the main text. Once again, the inverted softmax performs well when retrieving the Italian translations of English sentences, but is less effective translating Italian sentences into English. However, the performance of Dinu’s method appears to rise more rapidly than other methods as we transition from precision @1 to @5 to @10. Additionally, while Dinu’s method performs better when using the word dictionary @1, it prefers the phrase dictionary @5 and @10.

The paper focuses on bilingual word representation learning with the following setting:
1. Bilingual representation is learnt in an offline manner i.e., we already have monolingual representations for the source and target language and we are learning a common mapping for these two representations.
2. There is no direct word to word alignments available between the source and target language.
This is a practically useful setting to consider and authors have done a good job of unifying the existing solutions for this problem by providing theoretical justifications. Even though the authors do not propose a new method for offline bilingual representation learning, the paper is significant for the following contributions:
1. Theory for offline bilingual representation learning.
2. Inverted softmax.
3. Using cognate words for languages that share similar scripts.
4. Showing that this method also works at sentence level (to some extent).
Authors have addressed all my pre-review questions and I am ok with their response. I have few more comments:
1. Header for table 3 which says “word frequency” is misleading. “word frequency” could mean that rare words occur in row-1 while I guess authors meant to say that rare words occur in row-5.
2. I see that authors have removed precision @5 and @10 from table-6. Is it because of the space constraints or the results have different trend? I would like to see these results in the appendix.
3. In table-6 what is the difference between row-3 and row-4? Is the only difference NN vs. inverted softmax? Or there are other differences? Please elaborate.
4. Another suggestion is to try running an additional experiment where one can use both expert dictionary and cognate dictionary. Comparing all 3 methods in this setting should give more valuable insights about the usefulness of cognate dictionary.


#Exploration: A Study of Count-Based Exploration for Deep Reinforcement Learning
Count-based exploration algorithms are known to perform near-optimally when used in conjunction with tabular reinforcement learning (RL) methods for solving small discrete Markov decision processes (MDPs). It is generally thought that count-based methods cannot be applied in high-dimensional state spaces, since most states will only occur once. Recent deep RL exploration strategies are able to deal with high-dimensional continuous state spaces through complex heuristics, often relying on optimism in the face of uncertainty or intrinsic motivation. In this work, we describe a surprising finding: a simple generalization of the classic count-based approach can reach near state-of-the-art performance on various highdimensional and/or continuous deep RL benchmarks. States are mapped to hash codes, which allows to count their occurrences with a hash table. These counts are then used to compute a reward bonus according to the classic count-based exploration theory. We find that simple hash functions can achieve surprisingly good results on many challenging tasks. Furthermore, we show that a domain-dependent learned hash code may further improve these results. Detailed analysis reveals important aspects of a good hash function: 1) having appropriate granularity and 2) encoding information relevant to solving the MDP. This exploration strategy achieves near state-of-the-art performance on both continuous control tasks and Atari 2600 games, hence providing a simple yet powerful baseline for solving MDPs that require considerable exploration.
1 Introduction
Reinforcement learning (RL) studies an agent acting in an initially unknown environment, learning through trial and error to maximize rewards. It is impossible for the agent to act near-optimally until it has sufficiently explored the environment and identified all of the opportunities for high reward, in all scenarios. A core challenge in RL is how to balance exploration—actively seeking out novel states and actions that might yield high rewards and lead to long-term gains; and exploitation—maximizing short-term rewards using the agent’s current knowledge. While there are exploration techniques for finite MDPs that enjoy theoretical guarantees, there are no fully satisfying techniques for highdimensional state spaces; therefore, developing more general and robust exploration techniques is an active area of research.
Most of the recent state-of-the-art RL results have been obtained using simple exploration strategies such as uniform sampling (Mnih et al., 2015) and i.i.d./correlated Gaussian noise (Schulman et al., 2015; Lillicrap et al., 2015). Although these heuristics are sufficient in tasks with well-shaped rewards, the sample complexity can grow exponentially (with state space size) in tasks with sparse rewards (Osband et al., 2016b). Recently developed exploration strategies for deep RL have led to significantly improved performance on environments with sparse rewards. Bootstrapped DQN ∗These authors contributed equally.
(Osband et al., 2016a) led to faster learning in a range of Atari 2600 games by training an ensemble of Q-functions. Intrinsic motivation methods using pseudo-counts achieve state-of-the-art performance on Montezuma’s Revenge, an extremely challenging Atari 2600 game (Bellemare et al., 2016). Variational Information Maximizing Exploration (VIME, Houthooft et al. (2016)) encourages the agent to explore by acquiring information about environment dynamics, and performs well on various robotic locomotion problems with sparse rewards. However, we have not seen a very simple and fast method that can work across different domains.
Some of the classic, theoretically-justified exploration methods are based on counting state-action visitations, and turning this count into a bonus reward. In the bandit setting, the well-known UCB algorithm of Lai & Robbins (1985) chooses the action at at time t that maximizes r̂ (at ) + √
2 log t n(at )
where r̂ (at ) is the estimated reward, and n(at ) is the number of times action at was previously chosen. In the MDP setting, some of the algorithms have similar structure, for example, Model Based Interval Estimation–Exploration Bonus (MBIE-EB) of Strehl & Littman (2008) counts state-action pairs with a table n(s, a) and adding a bonus reward of the form β√
n(s,a) to encourage exploring less visited pairs.
Kolter & Ng (2009) show that the inverse-square-root dependence is optimal. MBIE and related algorithms assume that the augmented MDP is solved analytically at each timestep, which is only practical for small finite state spaces.
This paper presents a simple approach for exploration, which extends classic counting-based methods to high-dimensional, continuous state spaces. We discretize the state space with a hash function and apply a bonus based on the state-visitation count. The hash function can be chosen to appropriately balance generalization across states, and distinguishing between states. We select problems from rllab (Duan et al., 2016) and Atari 2600 (Bellemare et al., 2012) featuring sparse rewards, and demonstrate near state-of-the-art performance on several games known to be hard for naïve exploration strategies. The main strength of the presented approach is that it is fast, flexible and complementary to most existing RL algorithms.
In summary, this paper proposes a generalization of classic count-based exploration to high-dimensional spaces through hashing (Section 2); demonstrates its effectiveness on challenging deep RL benchmark problems and analyzes key components of well-designed hash functions (Section 3).
2 Methodology
2.1 Notation
This paper assumes a finite-horizon discounted Markov decision process (MDP), defined by (S,A,P, r, ρ0, γ,T ), in which S is the state space, A the action space, P a transition probability distribution, r : S × A → R≥0 a reward function, ρ0 an initial state distribution, γ ∈ (0, 1] a discount factor, and T the horizon. The goal of RL is to maximize the total expected discounted reward Eπ,P [∑T t=0 γ tr (st, at ) ] over a policy π, which outputs a distribution over actions given a state.
2.2 Count-Based Exploration via Static Hashing
Our approach discretizes the state space with a hash function φ : S → Z. An exploration bonus is added to the reward function, defined as
r+(s, a) = β√
n(φ(s)) , (1)
where β ∈ R≥0 is the bonus coefficient. Initially the counts n(·) are set to zero for the whole range of φ. For every state st encountered at time step t, n(φ(st )) is increased by one. The agent is trained with rewards (r + r+), while performance is evaluated as the sum of rewards without bonuses.
Note that our approach is a departure from count-based exploration methods such as MBIE-EB since we use a state-space count n(s) rather than a state-action count n(s, a). State-action counts n(s, a) are investigated in Appendix A.6, but no significant performance gains over state counting could be witnessed.
Algorithm 1: Count-based exploration through static hashing 1 Define state preprocessor g : S → RK 2 (In case of SimHash) Initialize A ∈ Rk×K with entries drawn i.i.d. from the standard Gaussian
distribution N (0, 1) 3 Initialize a hash table with values n(·) ≡ 0 4 for each iteration j do 5 Collect a set of state-action samples {(sm, am)}Mm=0 with policy π 6 Compute hash codes through any LSH method, e.g., for SimHash, φ(sm) = sgn(Ag(sm)) 7 Update the hash table counts ∀m : 0 ≤ m ≤ M as n(φ(sm)) ← n(φ(sm)) + 1
8 Update the policy π using rewards { r (sm, am) + β√
n(φ(sm )) }M m=0 with any RL algorithm
Clearly the performance of this method will strongly depend on the choice of hash function φ. One important choice we can make regards the granularity of the discretization: we would like for “distant” states to be be counted separately while “similar” states are merged. If desired, we can incorporate prior knowledge into the choice of φ, if there would be a set of salient state features which are known to be relevant.
Algorithm 1 summarizes our method. The main idea is to use locality-sensitive hashing (LSH) to convert continuous, high-dimensional data to discrete hash codes. LSH is a popular class of hash functions for querying nearest neighbors based on certain similarity metrics (Andoni & Indyk, 2006). A computationally efficient type of LSH is SimHash (Charikar, 2002), which measures similarity by angular distance. SimHash retrieves a binary code of state s ∈ S as
φ(s) = sgn(Ag(s)) ∈ {−1, 1}k, (2) where g : S → Rd is an optional preprocessing function and A is a k × d matrix with i.i.d. entries drawn from a standard Gaussian distributionN (0, 1). The value for k controls the granularity: higher values lead to fewer collisions and are thus more likely to distinguish states.
2.3 Count-Based Exploration via Learned Hashing
When the MDP states have a complex structure, as is the case with image observations, measuring their similarity directly in pixel space fails to provide the semantic similarity measure one would desire. Previous work in computer vision (Lowe, 1999; Dalal & Triggs, 2005; Tola et al., 2010) introduce manually designed feature representations of images that are suitable for semantic tasks including detection and classification. More recent methods learn complex features directly from data by training convolutional neural networks (Krizhevsky et al., 2012; Simonyan & Zisserman, 2014; He et al., 2015). Considering these results, it may be difficult for SimHash to cluster states appropriately using only raw pixels.
Therefore, we propose to use an autoencoder (AE) consisting of convolutional, dense, and transposed convolutional layers to learn meaningful hash codes in one of its hidden layers. This AE takes as input states s and contains one special dense layer comprised of K saturating activation functions,
Algorithm 2: Count-based exploration using learned hash codes 1 Define state preprocessor g : S → BK as the binary code resulting from the autoencoder (AE) 2 Initialize A ∈ Rk×K with entries drawn i.i.d. from the standard Gaussian distribution N (0, 1) 3 Initialize a hash table with values n(·) ≡ 0 4 for each iteration j do 5 Collect a set of state-action samples {(sm, am)}Mm=0 with policy π 6 Add the state samples {sm}Mm=0 to a FIFO replay pool R 7 if j mod jupdate = 0 then 8 Update the AE loss function in Eq. (3) using samples drawn from the replay pool
{sn}Nn=1 ∼ R, for example using stochastic gradient descent 9 Compute g(sm) = bb(sm)e, the K-dim rounded hash code for sm learned by the AE
10 Project g(sm) to a lower dimension k via SimHash as φ(sm) = sgn(Ag(sm)) 11 Update the hash table counts ∀m : 0 ≤ m ≤ M as n(φ(sm)) ← n(φ(sm)) + 1
12 Update the policy π using rewards { r (sm, am) + β√
n(φ(sm )) }M m=0 with any RL algorithm
more specifically sigmoid functions. By rounding the sigmoid output b(s) of this layer to the closest binary number, any state s can be binarized.
Since gradients cannot be back-propagated through a rounding function, an alternative method must be used to ensure that distinct states are mapped to distinct binary codes. Therefore, uniform noise U (−a, a) is added to the sigmoid output. By choosing uniform noise with a sufficiently high variance, the AE is only capable of reconstructing distinct inputs s if its hidden dense layer outputs values b(s) that are sufficiently far apart from each other (Gregor et al., 2016). Feeding a state s to the AE input, extracting b(s) and rounding it to bb(s)e yields a learned binary code. As such, the loss function L(·) over a set of collected states {si }Ni=1 is defined as
L ( {sn}Nn=1 ) = − 1
N N∑ n=1  log p(sn) − λ K K∑ i=1 min { (1 − bi (sn))2 , bi (sn)2 } . (3)
This objective function consists of a cross-entropy term and a term that pressures the binary code layer to take on binary values, scaled by λ ∈ R≥0. The reasoning behind this is that uniform noise U (−a, a) alone is insufficient, in case the AE does not use a particular sigmoid unit. This term ensures that an unused binary code output is assigned an arbitrary binary value. When omitting this term, the code is more prone to oscillations, causing unwanted bit flips, and destabilizing the counting process.
In order to make the AE train sufficiently fast—which is required since it is updated during the agent’s training—we make use of a pixel-wise softmax output layer (van den Oord et al., 2016) that shares weights between all pixels. The different softmax outputs merge together pixel intensities into discrete bins. The architectural details are described in Appendix A.1 and are depicted in Figure 1. Because the code dimension often needs to be large in order to correctly reconstruct the input, we apply a downsampling procedure to the resulting binary code bb(s)e, which can be done through random projection to a lower-dimensional space via SimHash as in Eq. (2).
One the one hand, it is important that the mapping from state to code needs to remain relatively consistent over time, which is nontrivial as the AE is constantly updated according to the latest data (Algorithm 2 line 8). An obvious solution would be to significantly downsample the binary code to a very low dimension, or by slowing down the training process. But on the other hand, the code has to remain relatively unique for states that are both distinct and close together on the image manifold. This is tackled both by the second term in Eq. (3) and by the saturating behavior of the sigmoid units. As such, states that are already well represented in the AE hidden layers tend to saturate the sigmoid units, causing the resulting loss gradients to be close to zero and making the code less prone to change.
3 Experiments
Experiments were designed to investigate and answer the following research questions:
1. Can count-based exploration through hashing improve performance significantly across different domains? How does the proposed method compare to the current state of the art in exploration for deep RL?
2. What is the impact of learned or static state preprocessing on the overall performance when image observations are used?
3. What factors contribute to good performance, e.g., what is the appropriate level of granularity of the hash function?
To answer question 1, we run the proposed method on deep RL benchmarks (rllab and ALE) that feature sparse rewards, and compare it to other state-of-the-art algorithms. Question 2 is answered by trying out different image preprocessors on Atari 2600 games. Finally, we investigate question 3 in Section 3.3 and 3.4. Trust Region Policy Optimization (TRPO, Schulman et al. (2015)) is chosen as the RL algorithm for all experiments, because it can handle both discrete and continuous action spaces, it can conveniently ensure stable improvement in the policy performance, and is relatively insensitive to hyperparameter changes. The hyperparameters settings are reported in Appendix A.1.
3.1 Continuous Control
The rllab benchmark (Duan et al., 2016) consists of various control tasks to test deep RL algorithms. We selected several variants of the basic and locomotion tasks that use sparse rewards, as shown in Figure 2, and adopt the experimental setup as defined in (Houthooft et al., 2016)—a description can be found in Appendix A.2. These tasks are all highly difficult to solve with naïve exploration strategies, such as adding Gaussian noise to the actions.
Figure 3 shows the results of TRPO (baseline), TRPO-SimHash, and VIME (Houthooft et al., 2016) on the classic tasks MountainCar and CartPoleSwingup, the locomotion task HalfCheetah, and the hierarchical task SwimmerGather. Using count-based exploration with hashing is capable of reaching the goal in all environments (which corresponds to a nonzero return), while baseline TRPO with Gaussian control noise fails completely. Although TRPO-SimHash picks up the sparse reward on HalfCheetah, it does not perform as well as VIME. In contrast, the performance of SimHash is comparable with VIME on MountainCar, while it outperforms VIME on SwimmerGather.
3.2 Arcade Learning Environment
The Arcade Learning Environment (ALE, Bellemare et al. (2012)), which consists of Atari 2600 video games, is an important benchmark for deep RL due to its high-dimensional state space and wide
variety of games. In order to demonstrate the effectiveness of the proposed exploration strategy, six games are selected featuring long horizons while requiring significant exploration: Freeway, Frostbite, Gravitar, Montezuma’s Revenge, Solaris, and Venture. The agent is trained for 500 iterations in all experiments, with each iteration consisting of 0.1 M steps (the TRPO batch size, corresponds to 0.4 M frames). Policies and value functions are neural networks with identical architectures to (Mnih et al., 2016). Although the policy and baseline take into account the previous four frames, the counting algorithm only looks at the latest frame.
BASS To compare with the autoencoder-based learned hash code, we propose using Basic Abstraction of the ScreenShots (BASS, also called Basic; see Bellemare et al. (2012)) as a static preprocessing function g. BASS is a hand-designed feature transformation for images in Atari 2600 games. BASS builds on the following observations specific to Atari: 1) the game screen has a low resolution, 2) most objects are large and monochrome, and 3) winning depends mostly on knowing object locations and motions. We designed an adapted version of BASS1, that divides the RGB screen into square cells, computes the average intensity of each color channel inside a cell, and assigns the resulting values to bins that uniformly partition the intensity range [0, 255]. Mathematically, let C be the cell size (width and height), B the number of bins, (i, j) cell location, (x, y) pixel location, and z the channel.
feature(i, j, z) = ⌊
B 255C2 ∑ (x,y)∈ cell(i, j) I (x, y, z) ⌋ . (4)
Afterwards, the resulting integer-valued feature tensor is converted to an integer hash code (φ(st ) in Line 6 of Algorithm 1). A BASS feature can be regarded as a miniature that efficiently encodes object locations, but remains invariant to negligible object motions. It is easy to implement and introduces little computation overhead. However, it is designed for generic Atari game images and may not capture the structure of each specific game very well.
We compare our results to double DQN (van Hasselt et al., 2016b), dueling network (Wang et al., 2016), A3C+ (Bellemare et al., 2016), double DQN with pseudo-counts (Bellemare et al., 2016), Gorila (Nair et al., 2015), and DQN Pop-Art (van Hasselt et al., 2016a) on the “null op” metric2. We show training curves in Figure 4 and summarize all results in Table 1. Surprisingly, TRPO-pixelSimHash already outperforms the baseline by a large margin and beats the previous best result on Frostbite. TRPO-BASS-SimHash achieves significant improvement over TRPO-pixel-SimHash on
1The original BASS exploits the fact that at most 128 colors can appear on the screen. Our adapted version does not make this assumption.
2The agent takes no action for a random number (within 30) of frames at the beginning of each episode.
Montezuma’s Revenge and Venture, where it captures object locations better than other methods.3 TRPO-AE-SimHash achieves near state-of-the-art performance on Freeway, Frostbite and Solaris.4
As observed in Table 1, preprocessing images with BASS or using a learned hash code through the AE leads to much better performance on Gravitar, Montezuma’s Revenge and Venture. Therefore, an static or adaptive preprocessing step can be important for a good hash function.
In conclusion, our count-based exploration method is able to achieve remarkable performance gains even with simple hash functions like SimHash on the raw pixel space. If coupled with domain-dependent state preprocessing techniques, it can sometimes achieve far better results.
3.3 Granularity
While our proposed method is able to achieve remarkable results without requiring much tuning, the granularity of the hash function should be chosen wisely. Granularity plays a critical role in count-based exploration, where the hash function should cluster states without under-generalizing or over-generalizing. Table 2 summarizes granularity parameters for our hash functions. In Table 3 we summarize the performance of TRPO-pixel-SimHash under different granularities. We choose Frostbite and Venture on which TRPO-pixel-SimHash outperforms the baseline, and choose as reward bonus coefficient β = 0.01 × 256k to keep average bonus rewards at approximately the same scale. k = 16 only corresponds to 65536 distinct hash codes, which is insufficient to distinguish between semantically distinct states and hence leads to worse performance. We observed that k = 512 tends to capture trivial image details in Frostbite, leading the agent to believe that every state is new and equally worth exploring. Similar results are observed while tuning the granularity parameters for TRPO-BASS-SimHash and TRPO-AE-SimHash.
The best granularity depends on both the hash function and the MDP. While adjusting granularity parameter, we observed that it is important to lower the bonus coefficient as granularity is increased. This is because a higher granularity is likely to cause lower state counts, leading to higher bonus rewards that may overwhelm the true rewards.
3We provide videos of example game play and visualizations of the difference bewteen Pixel-SimHash and BASS-SimHash at https://www.youtube.com/playlist?list=PLAd-UMX6FkBQdLNWtY8nH1-pzYJA_1T55
4Note that some design choices in other algorithms also impact exploration, such as ε-greedy and entropy regularization. Nevertheless, it is still valuable to position our results within the current literature.
3.4 A Case Study of Montezuma’s Revenge
Montezuma’s Revenge is widely known for its extremely sparse rewards and difficult exploration (Bellemare et al., 2016). While our method does not outperform Bellemare et al. (2016) on this game, we investigate the reasons behind this through various experiments. The experiment process below again demonstrates the importance of a hash function having the correct granularity and encoding relevant information for solving the MDP.
Our first attempt is to use game RAM states instead of image observations as inputs to the policy (details in Appendix A.1), which leads to a game score of 2500 with TRPO-BASS-SimHash. Our second attempt is to manually design a hash function that incorporates domain knowledge, called SmartHash, which uses an integer-valued vector consisting of the agent’s (x, y) location, room number and other useful RAM information as the hash code (details in Appendix A.3). The best SmartHash agent is able to obtain a score of 3500. Still the performance is not optimal. We observe that a slight change in the agent’s coordinates does not always result in a semantically distinct state, and thus the hash code may remain unchanged. Therefore we choose grid size s and replace the x coordinate by b(x − xmin)/sc (similarly for y). The bonus coefficient is chosen as β = 0.01 √ s to maintain the scale relative to the true reward5 (see Table 4). Finally, the best agent is able to obtain 6600 total rewards after training for 1000 iterations (1000 M time steps), with a grid size s = 10.
During our pursuit, we had another interesting discovery that the ideal hash function should not simply cluster states by their visual similarity, but instead by their relevance to solving the MDP. We
5The bonus scaling is chosen by assuming all states are visited uniformly and the average bonus reward should remain the same for any grid size.
experimented with including enemy locations in the first two rooms into SmartHash (s = 10), and observed that average score dropped to 1672 (at iteration 1000). Though it is important for the agent to dodge enemies, the agent also erroneously “enjoys” watching enemy motions at distance (since new states are constantly observed) and “forgets” that his main objective is to enter other rooms. An alternative hash function keeps the same entry “enemy locations”, but instead only puts randomly sampled values in it, which surprisingly achieves better performance (3112). However, by ignoring enemy locations altogether, the agent achieves a much higher score (5661) (see Figure 5). In retrospect, we examine the hash codes generated by BASS-SimHash and find that codes clearly distinguish between visually different states (including various enemy locations), but fails to emphasize that the agent needs to explore different rooms. Again this example showcases the importance of encoding relevant information in designing hash functions.
4 Related Work
Classic count-based methods such as MBIE (Strehl & Littman, 2005), MBIE-EB and (Kolter & Ng, 2009) solve an approximate Bellman equation as an inner loop before the agent takes an action (Strehl & Littman, 2008). As such, bonus rewards are propagated immediately throughout the state-action space. In contrast, contemporary deep RL algorithms propagate the bonus signal based on rollouts collected from interacting with environments, with value-based (Mnih et al., 2015) or policy gradient-based (Schulman et al., 2015; Mnih et al., 2016) methods, at limited speed. In addition, our proposed method is intended to work with contemporary deep RL algorithms, it differs from classical count-based method in that our method relies on visiting unseen states first, before the bonus reward can be assigned, making uninformed exploration strategies still a necessity at the beginning. Filling the gaps between our method and classic theories is an important direction of future research.
A related line of classical explorationmethods is based on the idea of optimism in the face of uncertainty (Brafman & Tennenholtz, 2002) but not restricted to using counting to implement “optimism”, e.g. R-Max (Brafman & Tennenholtz, 2002), UCRL (Jaksch et al., 2010), and E3 (Kearns & Singh, 2002). These methods, similar to MBIE and MBIE-EB, have theoretical guarantees in tabular settings.
Bayesian RL methods (Kolter & Ng, 2009; Guez et al., 2014; Sun et al., 2011; Ghavamzadeh et al., 2015), which keep track of a distribution over MDPs, are an alternative to optimism-based methods. Extensions to continuous state space have been proposed by Pazis & Parr (2013) and Osband et al. (2016b).
Another type of exploration is curiosity-based exploration. These methods try to capture the agent’s surprise about transition dynamics. As the agent tries to optimize for surprise, it naturally discovers novel states. We refer the reader to Schmidhuber (2010) and Oudeyer & Kaplan (2007) for an extensive review on curiosity and intrinsic rewards.
Several exploration strategies for deep RL have been proposed to handle high-dimensional state space recently. Houthooft et al. (2016) propose VIME, in which information gain is measured in Bayesian neural networks modeling the MDP dynamics, which is used an exploration bonus. Stadie et al. (2015) propose to use the prediction error of a learned dynamics model as an exploration bonus. Thompson sampling through bootstrapping is proposed by Osband et al. (2016a), using bootstrapped Q-functions.
The most related exploration strategy is proposed by Bellemare et al. (2016), in which an exploration bonus is added inversely proportional to the square root of a pseudo-count quantity. A state pseudocount is derived from its log-probability improvement according to a density model over the state space, which in the limit converges to the empirical count. Our method is similar to pseudo-count approach in the sense that both methods are performing approximate counting to have the necessary generalization over unseen states. The difference is that a density model has to be designed and learned to achieve good generalization for pseudo-count whereas in our case generalization is obtained by a wide range of simple hash functions (not necessarily SimHash). Another interesting connection is that our method also implies a density model ρ(s) = n(φ(s))N over all visited states, where N is the total number of states visited. Another method similar to hashing is proposed by Abel et al. (2016), which clusters states and counts cluster centers instead of the true states, but this method has yet to be tested on standard exploration benchmark problems.
5 Conclusions
This paper demonstrates that a generalization of classical counting techniques through hashing is able to provide an appropriate signal for exploration, even in continuous and/or high-dimensional MDPs using function approximators, resulting in near state-of-the-art performance across benchmarks. It provides a simple yet powerful baseline for solving MDPs that require informed exploration.
Acknowledgments
We would like to thank our colleagues at Berkeley and OpenAI for insightful discussions. This research was funded in part by ONR through a PECASE award. Yan Duan was also supported by a Berkeley AI Research lab Fellowship and a Huawei Fellowship. Xi Chen was also supported by a Berkeley AI Research lab Fellowship. We gratefully acknowledge the support of the NSF through grant IIS-1619362 and of the ARC through a Laureate Fellowship (FL110100281) and through the ARC Centre of Excellence for Mathematical and Statistical Frontiers. Adam Stooke gratefully acknowledges funding from a Fannie and John Hertz Foundation fellowship. Rein Houthooft is supported by a Ph.D. Fellowship of the Research Foundation - Flanders (FWO).
A Appendices
A.1 Hyperparameter Settings
For the rllab experiments, we used batch size 5000 for all tasks except SwimmerGather, for which we used batch size 50000. CartpoleSwingup makes use of a neural network policy with one layer of 32 tanh units. The other tasks make use of a two layer neural network policy of 32 tanh units each for MountainCar and HalfCheetah, and of 64 and 32 tanh units for SwimmerGather. The outputs are modeled by a fully factorized Gaussian distributionN (µ, σ2I), in which µ is modeled as the network output, while σ is a parameter. CartPoleSwingup makes use of a neural network baseline with one layer of 32 ReLU units, while all other tasks make use of a linear baseline function. For all tasks, we used TRPO step size 0.01 and discount factor γ = 0.99. We choose SimHash parameter k = 32 and bonus coefficient β = 0.01, found through a coarse grid search.
For Atari experiments, a batch size of 100000 is used, while the KL divergence step size is set to 0.01. The policy and baseline both have the following architecture: 2 convolutional layers with respectively 16 and 32 filters, sizes 8 × 8 and 4 × 4, strides 4 and 2, using no padding, feeding into a single hidden layer of 256 units. The nonlinearities are rectified linear units (ReLUs). The input frames are downsampled to 52 × 52. The input to policy and baseline consists of the 4 previous frames, corresponding to the frame skip of 4. The discount factor was set to γ = 0.995. All inputs are rescaled to [−1, 1] element-wise. All experiments used 5 different training seeds, except the experiments with the learned hash code, which uses 3 different training seeds. Batch normalization (Ioffe & Szegedy, 2015) is used at each policy and baseline layer. TRPO-pixel-SimHash uses binary codes of size k = 256; BASS (TRPO-BASS-SimHash) extracts features using cell size C = 20 and B = 20 bins. The autoencoder for the learned embedding (TRPO-AE-SimHash) uses a binary hidden layer of 512 bit, which are projected to 64 bit.
RAM states in Atari 2600 games are integer-valued vectors over length 128 in the range [0, 255]. Experiments on Montezuma’s Revenge with RAM observations use a policy consisting of 2 hidden layers, each of size 32. RAM states are rescaled to a range [−1, 1]. Unlike images, only the current RAM is shown to the agent. Experiment results are averaged over 10 random seeds.
In addition, we apply counting Bloom filters (Fan et al., 2000) to maintain a small hash table. Details can be found in Appendix A.5.
The autoencoder used for the learned hash code has a 512 bit binary code layer, using sigmoid units, to which uniform noise U (−a, a) with a = 0.3 is added. The loss function Eq. (3), using λ = 10, is updated every jupdate = 3 iterations. The architecture looks as follows: an input layer of size 52 × 52, representing the image luminance is followed by 3 consecutive 6 × 6 convolutional layers with stride 2 and 96 filters feed into a fully connected layer of size 1024, which connects to the binary code layer. This binary code layer feeds into a fully-connected layer of 1024 units, connecting to a fully-connected layer of 2400 units. This layer feeds into 3 consecutive 6× 6 transposed convolutional layers of which the final one connects to a pixel-wise softmax layer with 64 bins, representing the pixel intensities. Moreover, label smoothing is applied to the different softmax bins, in which the log-probability of each of the bins is increased by 0.003, before normalizing. The softmax weights are shared among each pixel. All output nonlinearities are ReLUs; Adam (Kingma & Ba, 2015) is used as an optimization scheme; batch normalization (Ioffe & Szegedy, 2015) is applied to each layer. The architecture was shown in Figure 1 of Section 2.3.
A.2 Description of the Adapted rllab Tasks
This section describes the continuous control environments used in the experiments. The tasks are implemented as described in Duan et al. (2016), following the sparse reward adaptation of Houthooft et al. (2016). The tasks have the following state and action dimensions: CartPoleSwingup, S ⊆ R4, A ⊆ R1; MountainCar S ⊆ R3, A ⊆ R1; HalfCheetah, S ⊆ R20, A ⊆ R6; SwimmerGather, S ⊆ R33, A ⊆ R2. For the sparse reward experiments, the tasks have been modified as follows. In CartPoleSwingup, the agent receives a reward of +1 when cos(β) > 0.8, with β the pole angle. In MountainCar, the agent receives a reward of +1 when the goal state is reached, namely escaping the valley from the right side. Therefore, the agent has to figure out how to swing up the pole in the absence of any initial external rewards. In HalfCheetah, the agent receives a reward of +1 when
xbody > 5. As such, it has to figure out how to move forward without any initial external reward. The time horizon is set to T = 500 for all tasks.
A.3 Examples of Atari 2600 RAM Entries
Table 5 lists the semantic interpretation of certain RAM entries in Montezuma’s Revenge. SmartHash, as described in Section 3.4, makes use of RAM indices 3, 42, 43, 27, and 67. “Beam walls” are deadly barriers that occur periodically in some rooms.
A.4 Analysis of Learned Binary Representation
Figure 6 shows the downsampled codes learned by the autoencoder for several Atari 2600 games (Frostbite, Freeway, and Montezuma’s Revenge). Each row depicts 50 consecutive frames (from 0 to 49, going from left to right, top to bottom). The pictures in the right column depict the binary codes that correspond with each of these frames (one frame per row). Figure 7 shows the reconstructions of several subsequent images according to the autoencoder.
A.5 Counting Bloom Filter/Count-Min Sketch
We experimented with directly building a hashing dictionary with keys φ(s) and values the state counts, but observed an unnecessary increase in computation time. Our implementation converts the integer hash codes into binary numbers and then into the “bytes” type in Python. The hash table is a dictionary using those bytes as keys.
However, an alternative technique called Count-Min Sketch (Cormode & Muthukrishnan, 2005), with a data structure identical to counting Bloom filters (Fan et al., 2000), can count with a fixed integer array and thus reduce computation time. Specifically, let p1, . . . , pl be distinct large prime numbers and define φ j (s) = φ(s) mod pj . The count of state s is returned as min1≤ j≤l n j ( φ j (s) ) . To increase
the count of s, we increment n j ( φ j (s) ) by 1 for all j. Intuitively, the method replaces φ by weaker
hash functions, while it reduces the probability of over-counting by reporting counts agreed by all such weaker hash functions. The final hash code is represented as ( φ1(s), . . . , φl (s) ) .
Throughout all experiments above, the prime numbers for the counting Bloom filter are 999931, 999953, 999959, 999961, 999979, and 999983, which we abbreviate as “6 M”. In addition, we experimented with 6 other prime numbers, each approximately 15 M, which we abbreviate as “90 M”. As we can see in Figure 8, counting states with a dictionary or with Bloom filters lead to similar performance, but the computation time of latter is lower. Moreover, there is little difference between direct counting and using a very larger table for Bloom filters, as the average bonus rewards are almost the same, indicating the same degree of exploration-exploitation trade-off. On the other hand, Bloom filters require a fixed table size, which may not be known beforehand.
Theory of Bloom Filters Bloom filters (Bloom, 1970) are popular for determining whether a data sample s′ belongs to a dataset D. Suppose we have l functions φ j that independently assign each data sample to an integer between 1 and p uniformly at random. Initially 1, 2, . . . , p are marked as 0. Then every s ∈ D is “inserted” through marking φ j (s) as 1 for all j. A new sample s′ is reported as a member of D only if φ j (s) are marked as 1 for all j. A bloom filter has zero false negative rate (any s ∈ D is reported a member), while the false positive rate (probability of reporting a nonmember as a member) decays exponentially in l.
Though Bloom filters support data insertion, it does not allow data deletion. Counting Bloom filters (Fan et al., 2000) maintain a counter n(·) for each number between 1 and p. Inserting/deleting s corresponds to incrementing/decrementing n ( φ j (s) ) by 1 for all j. Similarly, s is considered a
member if ∀ j : n ( φ j (s) ) = 0.
Count-Min sketch is designed to support memory-efficient counting without introducing too many over-counts. It maintains a separate count n j for each hash function φ j defined as φ j (s) = φ(s) mod pj , where pj is a large prime number. For simplicity, we may assume that pj ≈ p ∀ j and φ j assigns s to any of 1, . . . , p with uniform probability.
We now derive the probability of over-counting. Let s be a fixed data sample (not necessarily inserted yet) and suppose a dataset D of N samples are inserted. We assume that pl N . Let n := min1≤ j≤l n j ( φ j (s) ) be the count returned by the Bloom filter. We are interested in computing
Prob(n > 0|s < D). Due to assumptions about φ j , we know n j (φ(s)) ∼ Binomial ( N, 1p ) . Therefore,
Prob(n > 0|s < D) = Prob(n > 0, s < D) Prob(s < D)
= Prob(n > 0) − Prob(s ∈ D)
Prob(s < D)
≈ Prob(n > 0) Prob(s < D)
=
∏l j=1 Prob(n j (φ j (s)) > 0)
(1 − 1/pl)N
= (1 − (1 − 1/p)N )l
(1 − 1/pl)N
≈ (1 − e −N/p)l
e−N/pl
≈ (1 − e−N/p)l .
(5)
In particular, the probability of over-counting decays exponentially in l. We refer the readers to (Cormode & Muthukrishnan, 2005) for other properties of the Count-Min sketch.
A.6 Robustness Analysis
Apart from the experimental results shown in Table 1 and Table 3, additional experiments have been performed to study several properties of our algorithm.
Hyperparameter sensitivity To study the performance sensitivity to hyperparameter changes, we focus on evaluating TRPO-RAM-SimHash on the Atari 2600 game Frostbite, where the method has a clear advantage over the baseline. Because the final scores can vary between different random seeds, we evaluated each set of hyperparameters with 30 seeds. To reduce computation time and cost, RAM states are used instead of image observations.
128 – 1475 4248 2801 3239 3621 1543 395
256 – 2583 4497 4437 7849 3516 2260 374
The results are summarized in Table 6. Herein, k refers to the length of the binary code for hashing while β is the multiplicative coefficient for the reward bonus, as defined in Section 2.2. This table demonstrates that most hyperparameter settings outperform the baseline (β = 0) significantly. Moreover, the final scores show a clear pattern in response to changing hyperparameters. Small β-values lead to insufficient exploration, while large β-values cause the bonus rewards to overwhelm the true rewards. With a fixed k, the scores are roughly concave in β, peaking at around 0.2. Higher granularity k leads to better performance. Therefore, it can be concluded that the proposed exploration method is robust to hyperparameter changes in comparison to the baseline, and that the best parameter settings can obtained from a relatively coarse-grained grid search.
State and state-action counting Continuing the results in Table 6, the performance of state-action counting is studied using the same experimental setup, summarized in Table 7. In particular, a bonus reward r+ = β√
n(s,a) instead of r+ = β√ n(s) is assigned. These results show that the relative
performance of state counting compared to state-action counting depends highly on the selected hyperparameter settings. However, we notice that the best performance is achieved using state counting with k = 256 and β = 0.2.
128 1475 / 808 4248 / 4302 2801 / 4802 3239 / 7291 3621 / 4243 1543 / 1941 395 / 362

The paper proposes a new exploration scheme for reinforcement learning using locality-sensitive hashing states to build a table of visit counts which are then used to encourage exploration in the style of MBIE-EB of Strehl and Littman.
Several points are appealing about this approach: first, it is quite simple compared to the current alternatives (e.g. VIME, density estimation and pseudo-counts). Second, the paper presents results across several domains, including classic benchmarks, continuous control domains, and Atari 2600 games. In addition, there are results for comparison from several other algorithms (DQN variants), many of which are quite recent. The results indicate that the approach clearly improves over the baseline. The results against other exploration algorithms are not as clear (more dependent on the individual domain/game), but I think this is fine as the appeal of the technique is its simplicity. Third, the paper presents results on the sensitivity to the granularity of the abstraction.
I have only one main complaint, which is it seems there was some engineering involved to get this to work, and I do not have much confidence in the robustness of the conclusions. I am left uncertain as to how the story changes given slight perturbations over hyper-parameter values or enabling/disabling of certain choices. For example, how critical was using PixelCNN (or tying the weights?) or noisifying the output in the autoencoder, or what happens if you remove the custom additions to BASS? The granularity results show that the choice of resolution is sensitive, and even across games the story is not consistent.
The authors decide to use state-based counts instead of state-action based counts, deviating from the theory, which is odd because the reason to used LSH in the first place is to get closer to what MBIE-EB would advise via tabular counts. There are several explanations as to why state-based versus state-action based counts perform similarly in Atari; the authors do not offer any. Why?
It seems like the technique could be easily used in DQN as well, and many of the variants the authors compare to are DQN-based, so omitting DQN here again seems strange. The authors justify their choice of TRPO by saying it ensures safe policy improvement, though it is not clear that this is still true when adding these exploration bonuses.
The case study on Montezuma's revenge, while interesting, involves using domain knowledge and so does not really fit well with the rest of the paper.
So, in the end, simple and elegant idea to help with exploration tested in many domains, though I am not certain which of the many pieces are critical for the story to hold versus just slightly helpful, which could hurt the long-term impact of the paper.
--- After response:
Thank you for the thorough response, and again my apologies for the late reply.
I appreciate the follow-up version on the robustness of SimHash and state counting vs. state-action counting.
The paper addresses an important problem (exploration), suggesting a "simple" (compared to density estimation) counting method via hashing. It is a nice alternative approach to the one offered by Bellemare et al. If discussion among reviewers were possible, I would now try to assemble an argument to accept the paper. Specifically, I am not as concerned about beating the state of the art in Montezuma's as Reviewer3 as the merit of the current paper is one the simplicity of the hashing and on the wide comparison of domains vs. the baseline TRPO. This paper shows that we should not give up on simple hashing. There still seems to be a bunch of fiddly bits to get this to work, and I am still not confident that these results are easily reproducible. Nonetheless, it is an interesting new contrasting approach to exploration which deserves attention.
Not important for the decision: The argument in the rebuttal concerning DQN & A3C is a bit of a straw man. I did not mention anything at all about A3C, I strictly referred to DQN, which is less sensitive to parameter-tuning than A3C. Also, Bellemare 2016 main result on Montezuma used DQN. Hence the omission of these techniques applied to DQN still seems a bit strange (for the Atari experiments). The figure S9 from Mnih et al. points to instances of asynchronous one-step Sarsa with varied thread counts.. of course this will be sensitive to parameters: it is both asynchronous online algorithms *and* the parameter varied is the thread count! This is hardly indicative of DQN's sensitivity to parameters, since DQN is (a) single-threaded (b) uses experience replay, leading to slower policy changes. Another source of stability, DQN uses a target network that changes infrequently. Perhaps the authors made a mistake in the reference graph in the figure? (I see no Figure 9 in