DIVERSE BEAM SEARCH: DECODING DIVERSE SOLUTIONS FROM NEURAL SEQUENCE MODELS
Neural sequence models are widely used to model time-series data. Equally ubiquitous is the usage of beam search (BS) as an approximate inference algorithm to decode output sequences from these models. BS explores the search space in a greedy left-right fashion retaining only the top B candidates. This tends to result in sequences that differ only slightly from each other. Producing lists of nearly identical sequences is not only computationally wasteful but also typically fails to capture the inherent ambiguity of complex AI tasks. To overcome this problem, we propose Diverse Beam Search (DBS), an alternative to BS that decodes a list of diverse outputs by optimizing a diversity-augmented objective. We observe that our method not only improved diversity but also finds better top 1 solutions by controlling for the exploration and exploitation of the search space. Moreover, these gains are achieved with minimal computational or memory overhead compared to beam search. To demonstrate the broad applicability of our method, we present results on image captioning, machine translation, conversation and visual question generation using both standard quantitative metrics and qualitative human studies. We find that our method consistently outperforms BS and previously proposed techniques for diverse decoding from neural sequence models.
1 INTRODUCTION
In the last few years, Recurrent Neural Networks (RNNs), Long Short-Term Memory networks (LSTMs) or more generally, neural sequence models have become the standard choice for modeling time-series data for a wide range of applications including speech recognition (Graves et al., 2013), machine translation (Bahdanau et al., 2014), conversation modeling (Vinyals & Le, 2015), image and video captioning (Vinyals et al., 2015; Venugopalan et al., 2015), and visual question answering (Antol et al., 2015). RNN based sequence generation architectures model the conditional probability, Pr(y|x) of an output sequence y = (y1, . . . , yT ) given an input x (possibly also a sequence); where the output tokens yt are from a finite vocabulary, V . Inference in RNNs. Maximum a Posteriori (MAP) inference for RNNs is the task of finding the most likely output sequence given the input. Since the number of possible sequences grows as |V|T , exact inference is NP-hard – so, approximate inference algorithms like beam search (BS) are commonly employed. BS is a heuristic graph-search algorithm that maintains the B top-scoring partial sequences expanded in a greedy left-to-right fashion. Fig. 1 shows a sample BS search tree.
Lack of Diversity in BS. Despite the widespread usage of BS, it has long been understood that solutions decoded by BS are generic and lacking in diversity (Finkel et al., 2006; Gimpel et al.,
a
train
steam black locomotive
is traveling
on
engine train train
coming down a
the
train engine
down track train tracks traveling is the with near track down
through
tracks
a with train tracks a
in
a tracks
forest lush
a
an
train steam
an the
is engine
old train
a an
coming train
train steam
train black
traveling is
engine locomotive
train and
down through
train is
is white
train a
is traveling
coming on
tracks
forest
down through
the a
Beam Search
Diverse Beam Search
A steam engine train travelling down train tracks. A steam engine train travelling down tracks. A steam engine train travelling through a forest. A steam engine train travelling through a lush green forest. A steam engine train travelling through a lush green countryside A train on a train track with a sky background.
A steam engine travelling down train tracks. A steam engine train travelling through a forest. An old steam engine train travelling down train tracks. An old steam engine train travelling through a forest. A black train is on the tracks in a wooded area. A black train is on the tracks in a rural area.
Single engine train rolling down the tracks. A steam locomotive is blowing steam. A locomotive drives along the tracks amongst trees and bushes. An old fashion train with steam coming out of its pipe. A black and red train moving down a train track.
An engine is coming down the train track. Ground Truth Captions
Figure 1: Comparing image captioning outputs decoded by BS (top) and our method, Diverse Beam Search (middle) – we notice that BS captions are near-duplicates with similar shared paths in the search tree and minor variations in the end. In contrast, DBS captions are significantly diverse and similar to the variability in human-generated ground truth captions (bottom).
2013; Li et al., 2015; Li & Jurafsky, 2016). Comparing the human (bottom) and BS (top) generated captions shown in Fig. 1 demonstrates this deficiency. While this behavior of BS is disadvantageous for many reasons, we highlight the three most crucial ones here:
i) The production of near-identical beams make BS a computationally wasteful algorithm, with essentially the same computation being repeated for no significant gain in performance.
ii) Due to loss-evaluation mismatch (i.e. improvements in posterior-probabilities not necessarily corresponding to improvements in task-specific metrics), it is common practice to deliberately throttle BS to become a poorer optimization algorithm by using reduced beam widths (Vinyals et al., 2015; Karpathy & Fei-Fei, 2015; Ferraro et al., 2016). This treatment of an optimization algorithm as a hyperparameter is not only intellectually dissatisfying but also has a significant practical side-effect – it leads to the decoding of largely bland, generic, and “safe” outputs, e.g. always saying “I don’t know” in conversation models (Kannan et al., 2016).
iii) Most importantly, lack of diversity in the decoded solutions is fundamentally crippling in AI problems with significant ambiguity – e.g. there are multiple ways of describing an image or responding in a conversation that are “correct” and it is important to capture this ambiguity by finding several diverse plausible hypotheses.
Overview and Contributions. To address these shortcomings, we propose Diverse Beam Search (DBS) – a general framework to decode a set of diverse sequences that can be used as an alternative to BS. At a high level, DBS decodes diverse lists by dividing the given beam budget into groups and enforcing diversity between groups of beams. Drawing from recent work in the probabilistic graphical models literature on Diverse M-Best (DivMBest) MAP inference (Batra et al., 2012; Prasad et al., 2014; Kirillov et al., 2015), we optimize an objective that consists of two terms – the sequence likelihood under the model and a dissimilarity term that encourages beams across groups to differ. This diversity-augmented model score is optimized in a doubly greedy manner – greedily optimizing along both time (like BS) and groups (like DivMBest).
Our primary technical contribution is Diverse Beam Search, a doubly greedy approximate inference algorithm to decode diverse sequences from neural sequence models. We report results on image captioning, machine translation, conversations and visual question generation to demonstrate the broad applicability of DBS. Results show that DBS produces consistent improvements on both task-specific oracle and other diversity-related metrics while maintaining run-time and memory requirements similar to BS. We also evaluate human preferences between image captions generated by BS or DBS. Further experiments show that DBS is robust over a wide range of its parameter values and is capable of encoding various notions of diversity through different forms of the diversty term.
Overall, our algorithm is simple to implement and consistently outperforms BS in a wide range of domains without sacrificing efficiency. Our implementation is publicly available at https: //github.com/ashwinkalyan/dbs. Additionally, we provide an interactive demonstration of DBS for image captioning at http://dbs.cloudcv.org.
2 PRELIMINARIES: DECODING RNNS WITH BEAM SEARCH
We begin with a refresher on BS, before describing our generalization, Diverse Beam Search. For notational convenience, let [n] denote the set of natural numbers from 1 to n and let v[n] = [v1, . . . , vn]
ᵀ index the first n elements of a vector v ∈ Rm. The Decoding Problem. RNNs are trained to estimate the likelihood of sequences of tokens from a finite dictionary V given an input x. The RNN updates its internal state and estimates the conditional probability distribution over the next output given the input and all previous output tokens. We denote the logarithm of this conditional probability distribution over all tokens at time t as θ(yt) = log Pr(yt|yt−1, . . . , y1,x). To avoid notational clutter, we index θ(·) with a single variable yt, but it should be clear that it depends on all previous outputs, y[t−1]. We write the log probability of a partial solution (i.e. the sum of log probabilities of all tokens decoded so far) as Θ(y[t]) =∑ τ∈[t] θ(yτ ). The decoding problem is then the task of finding a sequence y that maximizes Θ(y).
As each output is conditioned on all the previous outputs, decoding the optimal length-T sequence in this setting can be viewed as MAP inference on a T -order Markov chain with nodes corresponding to output tokens at each time step. Not only does the size of the largest factor in such a graph grow as |V|T , but computing these factors also requires repetitively evaluating the sequence model. Thus, approximate algorithms are employed and the most prevalent method is beam search (BS).
Beam search is a heuristic search algorithm which stores the topB highest scoring partial candidates at each time step; where B is known as the beam width. Let us denote the set of B solutions held by BS at the start of time t as Y[t−1] = {y1,[t−1], . . . ,yB,[t−1]}. At each time step, BS considers all possible single token extensions of these beams given by the set Yt = Y[t−1] × V and retains the B highest scoring extensions. More formally, at each step the beams are updated as
Y[t] = argmax y1,[t],...,yB,[t]∈Yt ∑ b∈[B] Θ(yb,[t]) s.t. yi,[t] 6= yj,[t] ∀i 6= j. (1)
The above objective can be trivially maximized by sorting all B × |V| members of Yt by their log probabilities and selecting the top B. This process is repeated until time T and the most likely sequence is selected by ranking the B complete beams according to their log probabilities.
While this method allows for multiple sequences to be explored in parallel, most completions tend to stem from a single highly valued beam – resulting in outputs that are often only minor perturbations of a single sequence (and typically only towards the end of the sequences).
3 DIVERSE BEAM SEARCH: FORMULATION AND ALGORITHM
To overcome this, we augment the objective in Eq. 1 with a dissimilarity term ∆(Y[t]) that measures the diversity between candidate sequences, assigning a penalty ∆(Y[t])[c] to each possible sequence completion c ∈ V . Jointly optimizing this augmented objective for allB candidates at each time step is intractable as the number of possible solutions grows with |V|B (easily 1060 for typical language modeling settings). To avoid this, we opt for a greedy procedure that divides the beam budget B into G groups and promotes diversity between these groups. The approximation is doubly greedy – across both time and groups – so ∆(Y[t]) is constant with respect to other groups and we can sequentially optimize each group using regular BS. We now explain the specifics of our approach.
Diverse Beam Search. As joint optimization is intractable, we form G smaller groups of beams and optimize them sequentially. Consider a partition of the set of beams Y[t] into G smaller sets Y g[t], g∈[G] of B
′ = B/G beams each (we pick G to divide B). In the example shown in Fig. 2, B = 6 beams are divided into G = 3 differently colored groups containing B′ = 2 beams each.
Considering diversity only between groups, reduces the search space at each time step; however, inference remains intractable. To enforce diversity efficiently, we consider a greedy strategy that steps each group forward in time sequentially while considering the others fixed. Each group can then evaluate the diversity term with respect to the fixed extensions of previous groups, returning the search space to B′ × |V|. In the snapshot shown in Fig. 2, the third group is being stepped forward at time step t = 4 and the previous groups have already been completed. With this staggered beamfront, the diversity term of the third group can be computed using these completions. Here we use
hamming diversity, which adds diversity penalty -1 for each appearance of a possible extension word at the same time step in a previous group – ‘birds’, ‘the’, and ‘an’ in the example – and 0 to all other possible completions. We discuss other forms for the diversity function in Section 5.1.
As we optimize each group with the previous groups fixed, extending group g at time t amounts to a standard BS using dissimilarity augmented log probabilities and can be written as:
Y g[t] = argmax yg 1,[t] ,...,yg B′,[t]∈Y g t
∑ b∈[B′] Θ ( ygb,[t] ) + λ∆ ( g−1⋃ h=1 Y h[t] ) [ygb,t], (2)
s.t. λ ≥ 0, ygi,[t] 6= y g j,[t]∀i 6= j
where λ is scalar controlling the strength of the diversity term. The full procedure to obtain diverse sequences using our method, Diverse Beam Search (DBS), is presented in Algorithm 1. It consists of two main steps for each group at each time step –
1) augmenting the log probabilities of each possible extension with the diversity term computed from previously advanced groups (Algorithm 1, Line 5) and,
2) running one step of a smaller BS with B′ beams using the augmented log probabilities to extend the current group (Algorithm 1, Line 6).
Note that the first group (g = 1) is not ‘conditioned’ on other groups during optimization, so our method is guaranteed to perform at least as well as a beam search of size B′.
Algorithm 1: Diverse Beam Search 1 Perform a diverse beam search with G groups using a beam width of B 2 for t = 1, . . . T do
// perform one step of beam search for first group without diversity 3 Y 1[t] ← argmax(y11,[t],...,y1B′,[t]) ∑ b∈[B′] Θ(y 1 b,[t]) 4 for g = 2, . . . G do // augment log probabilities with diversity penalty 5 Θ(ygb,[t])← Θ(y g b,[t]) + λ∆( ⋃g−1 h=1 Y h [t])[y g b,t] b ∈ [B′],y g b,[t] ∈ Y g t and λ > 0 // perform one step of beam search for the group 6 Y g[t] ← argmax(yg
1,[t] ,...,yg B′,[t]
)∑ b∈[B′] Θ(y g b,[t]) s.t. yi,[t] 6= yj,[t] ∀i 6= j
7 Return set of B solutions, Y[T ] = ⋃G g=1 Y g [T ]
4 RELATED WORK
Diverse M-Best Lists. The task of generating diverse structured outputs from probabilistic models has been studied extensively (Park & Ramanan, 2011; Batra et al., 2012; Kirillov et al., 2015; Prasad et al., 2014). Batra et al. (2012) formalized this task for Markov Random Fields as the DivMBest problem and presented a greedy approach which solves for outputs iteratively, conditioning on previous solutions to induce diversity. Kirillov et al. (2015) show how these solutions can be found
jointly (non-greedily) for certain kinds of energy functions. The techniques developed by Kirillov are not directly applicable to decoding from RNNs, which do not satisfy the assumptions made.
Most related to our proposed approach is the work of Gimpel et al. (2013), who applied DivMBest to machine translation using beam search as a black-box inference algorithm. Specifically, in this approach, DivMBest knows nothing about the inner-workings of BS and simply makesB sequential calls to BS to generate B diverse solutions. This approach is extremely wasteful because BS is called B times, run from scratch every time, and even though each call to BS produces B solutions, only one solution is kept by DivMBest. In contrast, DBS avoids these shortcomings by integrating diversity within BS such that no beams are discarded. By running multiple beam searches in parallel and at staggered time offsets, we obtain large time savings making our method comparable to a single run of classical BS. One potential disadvantage of our method w.r.t. Gimpel et al. (2013) is that sentence-level diversity metrics cannot be incorporated in DBS since no group is complete when diversity is encouraged. However, as observed empirically by us and Li et al. (2015), initial words tend to disproportionally impact the diversity of the resultant sequences – suggesting that later words may not be important for diverse inference.
Diverse Decoding for RNNs. Efforts have been made by Li et al. (2015) and Li & Jurafsky (2016) to produce diverse decodings from recurrent models for conversation modeling and machine translation. Both of these works propose new heuristics for creating diverse M-Best lists and employ mutual information to re-rank lists of sequences. The latter achieves a goal separate from ours, which is simply to re-rank diverse lists.
Li & Jurafsky (2016) proposes a BS diversification heuristic that discourages beams from sharing common roots, implicitly resulting in diverse lists. Introducing diversity through a modified objective (as in DBS) rather than via a procedural heuristic provides easier generalization to incorporate different notions of diversity and control the exploration-exploitation trade-off as detailed in Section 5.1. Furthermore, we find that DBS outperforms the method of Li & Jurafsky (2016).
Li et al. (2015) introduced a novel decoding objective that maximizes mutual information between inputs and predicted outputs to penalize generic sequences. This operates on a principle orthogonal and complementary to DBS and Li & Jurafsky (2016). It works by penalizing utterances that are generally more frequent (diversity independent of input) rather than penalizing utterances that are similar to other utterances produced for the same input (diversity conditioned on input). Furthermore, the input-independent approach requires training a new language model for the target language while DBS just requires a diversity function ∆. Combination of these complementary techniques is left as interesting future work.
In other recent work, Wu et al. (2016) modify the beam search objective by introducing lengthnormalization to favor longer sequences and a coverage penalty that favors sequences that account for the complete input sequence. While the coverage term does not generalize to all neural sequence models, the length-normalization term can be implemented by modifying the joint-log-probability of each sequence. Although the goal of this method is not to produce diverse lists and hence not directly comparable, it is a complementary technique that can be used in conjunction with our diverse decoding method.
5 EXPERIMENTS
In this section, we evaluate our approach on image captioning, machine translation, conversation and visual question generation tasks to demonstrate both its effectiveness against baselines and its general applicability to any inference currently supported by beam search. We also analyze the effects of DBS parameters, explore human preferences for diversity, and discuss diversity’s importance in explaining complex images. We first explain the baselines and evaluations used in this paper.
Baselines & Metrics. Apart from classical beam search, we compare DBS with the diverse decoding method proposed in Li & Jurafsky (2016). We also compare against two other complementary decoding techniques proposed in Li et al. (2015) and Wu et al. (2016). Note that these two techniques are not directly comparable with DBS since the goal is not to produce diverse lists. We now provide a brief description of the comparisons mentioned:
- Li & Jurafsky (2016): modify BS by introducing an intra-sibling rank. For each partial solution, the set of |V| beam extensions are sorted and assigned intra-sibling ranks k ∈ [|V|] in order
of decreasing log probabilities, θt(yt). The log probability of an extension is then reduced in proportion to its rank, and continuations are re-sorted under these modified log probabilities to select the top B ‘diverse’ beam extensions.
- Li et al. (2015): train an additional unconditioned target sequence model U(y) and perform BS decoding on an augmented objective P (y|x)− λU(y), penalizing input-independent decodings.
- Wu et al. (2016) modify the beam-search objective by introducing length-normalization that favors longer sequences. The joint log-probability of completed sequences is divided by a factor, (5 + |y|)α/(5 + 1)α, where α ∈ [0, 1].
We compare to our own implementations of these methods as none are publicly available. Both Li & Jurafsky (2016) and Li et al. (2015) develop and use re-rankers to pick a single solution from the generated lists. Since we are interested in evaluating the quality of the generated lists and in isolating the gains due to diverse decoding, we do not implement any re-rankers, simply sorting by log-probability.
We evaluate the performance of the generated lists using the following two metrics:
- Oracle Accuracy: Oracle or top k accuracy w.r.t. some task-specific metric, such as BLEU (Papineni et al., 2002) or SPICE (Anderson et al., 2016), is the maximum value of the metric achieved over a list of k potential solutions. Oracle accuracy is an upper bound on the performance of any re-ranking strategy and thus measures the maximum potential of a set of outputs.
- Diversity Statistics: We count the number of distinct n-grams present in the list of generated outputs. Similar to Li et al. (2015), we divide these counts by the total number of words generated to bias against long sentences.
Simultaneous improvements in both metrics indicate that output sequences have increased diversity without sacrificing fluency and correctness with respect to target tasks.
5.1 SENSITIVITY ANALYSIS AND EFFECT OF DIVERSITY FUNCTIONS
Here we discuss the impact of the number of groups, strength of diversity , and various forms of diversity for language models. Note that the parameters of DBS (and other baselines) were tuned on a held-out validation set for each experiment. The supplement provides further discussion and experimental details.
Number of Groups (G). Setting G=B allows for the maximum exploration of the search space, while settingG=1 reduces DBS to BS, resulting in increased exploitation of the search-space around the 1-best decoding. Empirically, we find that maximum exploration correlates with improved oracle accuracy and hence use G=B to report results unless mentioned otherwise. See the supplement for a comparison and more details.
Diversity Strength (λ). The diversity strength λ specifies the trade-off between the model score and diversity terms. As expected, we find that a higher value of λ produces a more diverse list; however, very large values of λ can overpower model score and result in grammatically incorrect outputs. We set λ via grid search over a range of values to maximize oracle accuracies achieved on the validation set. We find a wide range of λ values (0.2 to 0.8) work well for most tasks and datasets.
Choice of Diversity Function (∆). In Section 3, we defined ∆(·) as a function over a set of partial solutions that outputs a vector of dissimilarity scores for all possible beam completions. Assuming that each of the previous groups influences the completion of the current group independently, we can simplify ∆( ⋃g−1 h=1 Y h [t]) as the sum of each group’s contributions as ∑g−1 h=1 ∆(Y h [t]). In Section 3, we illustrated a simple hamming diversity of this form that penalizes selection of tokens proportionally to the number of time it was used in previous groups. However, this factorized diversity term can take various forms in our model – with hamming diversity being the simplest. For language models, we study the effect of using cumulative (i.e. considering all past time steps), n-gram and neural embedding based diversity functions. Each of these forms encode differing notions of diversity and result in DBS outperforming BS. We find simple hamming distance to be effective and report results based on this diversity measure unless otherwise specified. More details about these forms of the diversity term are provided in the supplementary.
5.2 IMAGE CAPTIONING
Dataset and Models. We evaluate on two datasets – COCO (Lin et al., 2014) and PASCAL-50S (Vedantam et al., 2015). We use the public splits as in Karpathy & Fei-Fei (2015) for COCO. PASCAL-50S is used only for testing (with 200 held out images used to tune hyperparameters). We train a captioning model (Vinyals et al., 2015) using the neuraltalk21 code repository.
Results. Table 1 shows Oracle (top k) SPICE for different values of k. DBS consistently outperforms BS and Li & Jurafsky (2016) on both datasets. We observe that gains on PASCAL-50S are more pronounced (7.14% and 4.65% SPICE@20 improvements over BS and Li & Jurafsky (2016)) than COCO. This suggests diverse predictions are especially advantageous when there is a mismatch between training and testing sets, implying DBS may be better suited for real-world applications.
Table 1 also shows the number of distinct n-grams produced by different techniques. Our method produces significantly more distinct n-grams (almost 300% increase in the number of 4-grams produced) as compared to BS. We also note that our method tends to produce slightly longer captions compared on average. Moreover, on the PASCAL-50S test split we observe that DBS finds more likely top-1 solutions on average – DBS obtains an average maximum log probability of -6.53 opposed to -6.91 found by BS of the same beam width. This empirical evidence suggests that using DBS as a replacement to BS may lead to lower inference approximation error.
Human Studies. To evaluate human preference between captions generated by DBS and BS, we perform a human study via Amazon Mechanical Turk using all 1000 images of PASCAL-50S. For each image, both DBS and standard BS captions are shown to 5 different users. They are then asked – “Which of the two robots understands the image better?” In this forced-choice test, DBS captions were preferred over BS 60% of the time by human annotators.
Is diversity always needed? While these results show that diverse outputs are important for systems that interact with users, is diversity always beneficial? While images with many objects (e.g., a park or a living room) can be described in multiple ways, the same is not true when there are few objects (e.g., a close up of a cat or a selfie). This notion is studied by Ionescu et al. (2016), which defines a “difficulty score”: the human response time for solving a visual search task. On the PASCAL50S dataset, we observe a positive correlation (ρ = 0.73) between difficulty scores and humans preferring DBS to BS. Moreover, while DBS is generally preferred by humans for ‘difficult’ images, both are about equally preferred on ‘easier’ images. Details are provided in the supplement.
5.3 MACHINE TRANSLATION
We use the WMT’14 dataset containing 4.5M sentences to train our machine translation models. We train stacking LSTM models as detailed in Luong et al. (2015), consisting of 4 layers and 1024- dimensional hidden states. While decoding sentences, we employ the same strategy to replace UNK tokens. We train our models using the publicly available seq2seq-attn2 code repository. We report results on news-test-2013 and news-test-2014 and use the news-test-2012 to tune the parameters of DBS. We use sentence level BLEU scores to compute oracle metrics and report distinct n-grams
1https://github.com/karpathy/neuraltalk2 2https://github.com/harvardnlp/seq2seq-attn
similar to image captioning. Results are shown in Table 2 and we again find that DBS consistently outperforms all baselines.
5.4 DIALOG GENERATION
Dialog generation is a task that is inherently diverse as there are multiple valid ways to respond to a statement. We train a seq2seq model consisting of LSTMs as in Vinyals & Le (2015) on the Cornell Movie Dialogs Corpus (Danescu-Niculescu-Mizil & Lee, 2011) using the neuralconvo3 repository. The training dataset consists of 222,182 conversational exchanges between movie characters. Since automatic evaluation of dialog generation responses is an open research problem with existing metrics being poorly correlated with human judgement (Liu et al., 2016), we show qualitative results to demonstrate the effectiveness of DBS. Table 3 compares BS and DBS at B=3.
5.5 VISUAL QUESTION GENERATION
We also report results on Visual Question Generation (VQG) (Mostafazadeh et al., 2016), where a model is trained to produce questions about an image. Generating visually focused questions is interesting because it requires reasoning about multiple problems that are central to vision – e.g., object attributes, relationships between objects, and natural language. Furthermore, many questions could make sense for one image, so it is important that lists of generated questions be diverse.
We use the VQA dataset (Antol et al., 2015) to train a model similar to image captioning architectures. Instead of captions, the training set now consists of 3 questions per image. Similar to previous results, using beam search to sample outputs results in similarly worded questions (see Fig. 3) and DBS brings out new details captured by the model. Counting the number of types of questions generated (as defined by Antol et al. (2015)) allows us to measure this diversity. We observe that the number of question types generated per image increases from 2.3 for BS to 3.7 for DBS (at B = 6).
6 CONCLUSION
Beam search is widely a used approximate inference algorithm for decoding sequences from neural sequence models; however, it suffers from a lack of diversity. Producing multiple highly similar and generic outputs is not only wasteful in terms of computation but also detrimental for tasks with
3https://github.com/macournoyer/neuralconvo
inherent ambiguity like many involving language. In this work, we modify Beam Search with a diversity-augmented sequence decoding objective to produce Diverse Beam Search. We develop a ‘doubly greedy’ approximate algorithm to minimize this objective and produce diverse sequence decodings. Our method consistently outperforms beam search and other baselines across all our experiments without extra computation or task-specific overhead. DBS is task-agnostic and can be applied to any case where BS is used, which we demonstrate in multiple domains. Our implementation available at https://github.com/ashwinkalyan/dbs.
APPENDIX
SENSIVITY STUDIES
Number of Groups. Fig. 4 presents snapshots of the transition from BS to DBS at B = 6 and G = {1, 3, 6}. As beam width moves from 1 to G, the exploration of the method increases resulting in more diverse lists.
Diversity Strength. As noted in Section 5.1, our method is robust to a wide range of values of the diversity strength (λ). Fig. 5a shows a grid search of λ for image-captioning on the PASCAL-50S dataset.
Choice of Diversity Function. The diversity function can take various forms ranging from simple hamming diversity to neural embedding based diversity. We discuss some forms for language modelling below:
- Hamming Diversity. This form penalizes the selection of tokens used in previous groups proportional to the number of times it was selected before.
- Cumulative Diversity. Once two sequences have diverged sufficiently, it seems unnecessary and perhaps harmful to restrict that they cannot use the same words at the same time. To encode this ‘backing-off’ of the diversity penalty we introduce cumulative diversity which keeps a count of identical words used at every time step, indicative of overall dissimilarity. Specifically, ∆(Y h[t])[y g [t]] = exp{−( ∑ τ∈t ∑ b∈B′ I[y h b,τ 6=y g b,τ ])/Γ} where Γ is a temperature parameter control-
ling the strength of the cumulative diversity term and I[·] is the indicator function. - n-gram Diversity. The current group is penalized for producing the same n-grams as previous
groups, regardless of alignment in time – similar to Gimpel et al. (2013). This is proportional to the number of times each n-gram in a candidate occurred in previous groups. Unlike hamming diversity, n-grams capture higher order structures in the sequences.
- Neural-embedding Diversity. While all the previous diversity functions discussed above perform exact matches, neural embeddings such as word2vec (Mikolov et al., 2013) can penalize semantically similar words like synonyms. This is incorporated in each of the previous diversity functions by replacing the hamming similarity with a soft version obtained by computing the cosine similarity between word2vec representations. When using with n-gram diversity, the representation of the n-gram is obtained by summing the vectors of the constituent words.
Each of these various forms encode different notions of diversity. Hamming diversity ensures different words are used at different times, but can be circumvented by small changes in sequence alignment. While n-gram diversity captures higher order statistics, it ignores sentence alignment. Neural-embedding based encodings can be seen as a semantic blurring of either the hamming or n-gram metrics, with word2vec representation similarity propagating diversity penalties not only to exact matches but also to close synonyms. Fig. 5b shows the oracle performace of various forms of the diversity function described in Section 5.1. We find that using any of the above functions help outperform BS in the tasks we examine; hamming diversity achieves the best oracle performance despite its simplicity.
IMAGE CAPTIONING EVALUATION
While we report oracle SPICE values in the paper, our method consistently outperforms baselines and classical BS on other standard metrics such as CIDEr (Table 4), METEOR (Table 5) and ROUGE (Table 6). We provide these additional results in this section.
Dataset Method Oracle Accuracy (CIDEr)
Modified SPICE evaluation. To measure both the quality and the diversity of the generated captions, we compute SPICE-score by comparing the graph union of all the generated hypotheses with the ground truth scene graph. This measure rewards all the relevant relations decoded as against oracle accuracy that compares to relevant relations present only in the top-scoring caption. We observe that DBS outperforms both baselines under this measure with a score of 18.345 as against a score of 16.988 (beam search) and 17.452 (Li & Jurafsky, 2016).
HUMAN STUDIES
For image-captioning, we conduct a human preference study between BS and DBS captions as explained in Section 5. A screen shot of the interface used to collect human preferences for captions generated using DBS and BS is presented in Fig. 6. The lists were shuffled to guard the task from being gamed by a turker.
As mentioned in Section 5, we observe that difficulty score of an image and human preference for DBS captions are positively correlated. The dataset contains more images that are less difficulty and so, we analyze the correlation by dividing the data into three bins. For each bin, we report the % of images for which DBS captions were preferred after a majority vote (i.e. at least 3/5 turkers voted in favor of DBS) in Table 7. At low difficulty scores consisting mostly of iconic images – one might expect that BS would be preferred more often than chance. However, mismatch between the statistics of the training and testing data results in a better performance of DBS. Some examples for this case are provided in Fig. 7. More general qualitative examples are provided in Fig. 8.
DISCUSSION
Are longer sentences better? Many recent works propose a scoring or a ranking objective that depends on the sequence length. These favor longer sequences, reasoning that they tend to have more details and resulting in improved accuracies. We measure the correlation between length of a sequence and its accuracy (here, SPICE) and observe insignificant correlation between SPICE and sequence length. On the PASCAL-50S dataset, we find that BS and DBS have are negatively correlated (ρ = −0.003 and ρ = −0.015 respectively), while (Li & Jurafsky, 2016) is correlated positively (ρ = 0.002). Length is not correlated with performance in this case.
Efficient utilization of beam budget. In this experiment, we emperically show that DBS makes efficient use of the beam budget in exploring the search space for better solutions. Fig. 9 shows the variation of oracle SPICE (@B) with the beam size. At really high beam widths, all decoding techniques achieve similar oracle accuracies. However, diverse decoding techniques like DBS achieve the same oracle at much lower beam widths. Hence, DBS not only produces sequence lists that are significantly different but also efficiently utilizes the beam budget to decode better solutions.

This paper considers the problem of decoding diverge solutions from neural sequence models. It basically adds an additional term to the log-likelihood of standard neural sequence models, and this additional term will encourage the solutions to be diverse. In addition to solve the inference, this paper uses a modified beam search.
On the plus side, there is not much work on producing diverse solutions in RNN/LSTM models. This paper represents one of the few works on this topic. And this paper is well-written and easy to follow.
The novel of this paper is relatively small. There has been a lot of prior work on producing diverse models in the area of probailistic graphical models. Most of them introduce an additional term in the objective function to encourage diversity. From that perspective, the solution proposed in this paper is not that different from previous work. Of course, one can argue that most previous work focues on probabilistic graphical models, while this paper focuses on RNN/LSTM models. But since RNN/LSTM can be simply interpreted as a probabilistic model, I would consider it a small novelty.
The diverse beam search seems to straightforward, i.e. it partitions the beam search space into groups, and does not consider the diversity within group (in order to reduce the search space). To me, this seems to be a simple trick. Note most previous work on diverse solutions in probabilistic graphical models usually involve developing some nontrivial algorithmic solutions, e.g. in order to achieve efficiency. In comparison, the proposed solution in this paper seems to be simplistic for a paper.
The experimental results how improvement over previous methods (Li & Jurafsky, 2015, 2016). But it is hard to say how rigorous the comparisons are, since they are based on the authors' own implementation of (Li & Jurasky, 2015, 2016).
---------------
update: given that the authors made the code available (I do hope the code will remain publicly available), this has alleviated some of my concerns about the rigor of the experiments. I will raise my rate to 6.


The current mainstream approach to train natural language systems is to expose them to large amounts of text. This passive learning is problematic if we are interested in developing interactive machines, such as conversational agents. We propose a framework for language learning that relies on multi-agent communication. We study this learning in the context of referential games. In these games, a sender and a receiver see a pair of images. The sender is told one of them is the target and is allowed to send a message from a fixed, arbitary vocabulary to the receiver. The receiver must rely on this message to identify the target. Thus, the agents develop their own language interactively out of the need to communicate. We show that two networks with simple configurations are able to learn to coordinate in the referential game. We further explore how to make changes to the game environment to cause the “word meanings” induced in the game to better reflect intuitive semantic properties of the images. In addition, we present a simple strategy for grounding the agents’ code into natural language. Both of these are necessary steps towards developing machines that are able to communicate with humans productively.
1 INTRODUCTION
I tried to break it to him gently [...] the only way to learn an unknown language is to interact with a native speaker [...] asking questions, holding a conversation, that sort of thing [...] If you want to learn the aliens’ language, someone [...] will have to talk with an alien. Recordings alone aren’t sufficient. Ted Chiang, Story of Your Life
One of the main aims of AI is to develop agents that can cooperate with others to achieve goals (Wooldridge, 2009). Such coordination requires communication. If the coordination partners are to include humans, the most obvious channel of communication is natural language. Thus, handling natural-language-based communication is a key step toward the development of AI that can thrive in a world populated by other agents.
Given the success of deep learning models in related domains such as image captioning or machine translation (e.g., Sutskever et al., 2014; Xu et al., 2015), it would seem reasonable to cast the problem of training conversational agents as an instance of supervised learning (Vinyals & Le, 2015). However, training on “canned” conversations does not allow learners to experience the interactive aspects of communication. Supervised approaches, which focus on the structure of language, are an excellent way to learn general statistical associations between sequences of symbols. However, they do not capture the functional aspects of communication, i.e., that humans use words to coordinate with others and make things happen (Austin, 1962; Clark, 1996; Wittgenstein, 1953).
This paper introduces the first steps of a research program based on multi-agent coordination communication games. These games place agents in simple environments where they need to develop a language to coordinate and earn payoffs. Importantly, the agents start as blank slates, but, by playing a game together, they can develop and bootstrap knowledge on top of each others, leading to the emergence of a language.
∗Work done while at Facebook AI Research.
The central problem of our program, then, is the following: How do we design environments that foster the development of a language that is portable to new situations and to new communication partners (in particular humans)?
We start from the most basic challenge of using a language in order to refer to things in the context of a two-agent game. We focus on two questions. First, whether tabula rasa agents succeed in communication. Second, what features of the environment lead to the development of codes resembling human language.
We assess this latter question in two ways. First, we consider whether the agents associate general conceptual properties, such as broad object categories (as opposed to low-level visual properties), to the symbols they learn to use. Second, we examine whether the agents’ “word usage” is partially interpretable by humans in an online experiment.
Other researchers have proposed communication-based environments for the development of coordination-capable AI. Work in multi-agent systems has focused on the design of pre-programmed communication systems to solve specific tasks (e.g., robot soccer, Stone & Veloso 1998). Most related to our work, Sukhbaatar et al. (2016) and Foerster et al. (2016) show that neural networks can evolve communication in the context of games without a pre-coded protocol. We pursue the same question, but further ask how we can change our environment to make the emergent language more interpretable.
Others (e.g., the SHRLDU program of Winograd 1971 or the game in Wang et al. 2016) propose building a communicating AI by putting humans in the loop from the very beginning. This approach has benefits but faces serious scalability issues, as active human intervention is required at each step. An attractive component of our game-based paradigm is that humans may be added as players, but do not need to be there all the time.
A third branch of research focuses on “Wizard-of-Oz” environments, where agents learn to play games by interacting with a complex scripted environment (Mikolov et al., 2015). This approach gives the designer tight control over the learning curriculum, but imposes a heavy engineering burden on developers. We also stress the importance of the environment (game setup), but we focus on simpler environments with multiple agents that force them to get smarter by bootstrapping on top of each other.
We leverage ideas from work in linguistics, cognitive science and game theory on the emergence of language (Wagner et al., 2003; Skyrms, 2010; Crawford & Sobel, 1982; Crawford, 1998). Our game is a variation of Lewis’ signaling game (Lewis, 1969). There is a rich tradition of linguistic and cognitive studies using similar setups (e.g., Briscoe, 2002; Cangelosi & Parisi, 2002; Spike et al., 2016; Steels & Loetzsch, 2012). What distinguishes us from this literature is our aim to, eventually, develop practical AI. This motivates our focus on more realistic input data (a large collection of noisy natural images) and on trying to align the agents’ language with human intuitions.
Lewis’ classic games have been studied extensively in game theory under the name of “cheap talk”. These games have been used as models to study the evolution of language both theoretically and experimentally (Crawford, 1998; Blume et al., 1998; Crawford & Sobel, 1982). A major question in game theory is whether equilibrium actually occurs in a game as convergence in learning is not guaranteed (Fudenberg & Peysakhovich, 2014; Roth & Erev, 1995). And, if an equilibrium is reached, which one it will be (since they are typically not unique). This is particularly true for cheap talk games, which exhibit Nash equilibria in which precise language emerges, others where vague language emerges and others where no language emerges at all (Crawford & Sobel, 1982). In addition, because in these games language has no ex-ante meaning and only emerges in the context of the equilibrium, some of the emergent languages may not be very natural. Our results speak to both the convergence question and the question of what features of the game cause the appearance of different types of languages. Thus, our results are also of interest to game theorists.
An evolutionary perspective has recently been advocated as a way to mitigate the data hunger of traditional supervised approaches (Goodfellow et al., 2014; Silver et al., 2016). This research confirms that learning can be bootstrapped from competition between agents. We focus, however, on cooperation between agents as a way to foster learning while reducing the need for annotated data.
2 GENERAL FRAMEWORK
Our general framework includes K players, each parametrized by θk, a collection of tasks/games that the players have to perform, a communication protocol V that enables the players to communicate with each other, and payoffs assigned to the players as a deterministic function of a well-defined goal. In this paper we focus on a particular version of this: referential games. These games are structured as follows.
1. There is a set of images represented by vectors {i1, . . . , iN}, two images are drawn at random from this set, call them (iL, iR), one of them is chosen to be the “target” t ∈ {L,R}
2. There are two players, a sender and a receiver, each seeing the images - the sender receives input θS(iL, iR, t)
3. There is a vocabulary V of size K and the sender chooses one symbol to send to the receiver, we call this the sender’s policy s(θS(iL, iR, t)) ∈ V
4. The receiver does not know the target, but sees the sender’s symbol and tries to guess the target image. We call this the receiver’s policy r(iL, iR, s(θS(iL, iR, t))) ∈ {L,R}
5. If r(iL, iR, s(θS(iL, iR, t)) = t, that is, if the receiver guesses the target, both players receive a payoff of 1 (win), otherwise they receive a payoff of 0 (lose).
Many extensions to the basic referential game explored here are possible. There can be more images, or a more sophisticated communication protocol (e.g., communication of a sequence of symbols or multi-step communication requiring back-and-forth interaction1), rotation of the sender and receiver roles, having a human occasionally playing one of the roles, etc.
3 EXPERIMENTAL SETUP
Images We use the McRae et al.’s (2005) set of 463 base-level concrete concepts (e.g., cat, apple, car. . . ) spanning across 20 general categories (e.g., animal, fruit/vegetable, vehicle. . . ). We randomly sample 100 images of each concept from ImageNet (Deng et al., 2009). To create target/distractor pairs, we randomly sample two concepts, one image for each concept and whether the first or second image will serve as target. We apply to each image a forward-pass through the pretrained VGG ConvNet (Simonyan & Zisserman, 2014), and represent it with the activations from either the top 1000-D softmax layer (sm) or the second-to-last 4096-D fully connected layer (fc).
Agent Players Both sender and receiver are simple feed-forward networks. For the sender, we experiment with the two architectures depicted in Figure 1. Both sender architectures take as input the target (marked with a green square in Figure 1) and distractor representations, always in this order, so that they are implicitly informed of which image is the target (the receiver, instead, sees the two images in random order).
The agnostic sender is a generic neural network that maps the original image vectors onto a “gamespecific” embedding space (in the sense that the embedding is learned while playing the game) followed by a sigmoid nonlinearity. Fully-connected weights are applied to the embedding concatenation to produce scores over vocabulary symbols.
The informed sender also first embeds the images into a “game-specific” space. It then applies 1-D convolutions (“filters”) on the image embeddings by treating them as different channels. The informed sender uses convolutions with kernel size 2x1 applied dimension-by-dimension to the two image embeddings (in Figure 1, there are 4 such filters). This is followed by the sigmoid nonlinearity. The resulting feature maps are combined through another filter (kernel size fx1, where f is the number of filters on the image embeddings), to produce scores for the vocabulary symbols. Intuitively, the informed sender has an inductive bias towards combining the two images dimensionby-dimension whereas the agnostic sender does not (though we note the agnostic architecture nests the informed one).
1For example, Jorge et al. (2016) explore agents playing a “Guess Who” game to learn about the emergence of question-asking and answering in language.
For both senders, motivated by the discrete nature of language, we enforce a strong communication bottleneck that discretizes the communication protocol. Activations on the top (vocabulary) layer are converted to a Gibbs distribution (with temperature parameter τ ), and then a single symbol s is sampled from the resulting probability distribution.
The receiver takes as input the target and distractor image vectors in random order, as well as the symbol produced by the sender (as a one-hot vector over the vocabulary). It embeds the images and the symbol into its own “game-specific” space. It then computes dot products between the symbol and image embeddings. Ideally, dot similarity should be higher for the image that is better denoted by the symbol. The two dot products are converted to a Gibbs distribution (with temperature τ ) and the receiver “points” to an image by sampling from the resulting distribution.
General Training Details We set the following hyperparameters without tuning: embedding dimensionality: 50, number of filters applied to embeddings by informed sender: 20, temperature of Gibbs distributions: 10. We explore two vocabulary sizes: 10 and 100 symbols.
The sender and receiver parameters θ = 〈θR, θS〉 are learned while playing the game. No weights are shared and the only supervision used is communication success, i.e., whether the receiver pointed at the right referent.
This setup is naturally modeled with Reinforcement Learning (Sutton & Barto, 1998). As outlined in Section 2, the sender follows policy s(θS(iL, iR, t)) ∈ V and the receiver policy r(iL, iR, s(θS(iL, iR, t))) ∈ {L,R}. The loss function that the two agents must minimize is −IEr̃[R(r̃)] where R is the reward function returning 1 iff r(iL, iR, s(θS(iL, iR, t)) = t. Parameters are updated through the Reinforce rule (Williams, 1992). We apply mini-batch updates, with a batch size of 32 and for a total of 50k iterations (games). At test time, we compile a set of 10k games using the same method as for the training games.
We now turn to our main questions. The first is whether the agents can learn to successfully coordinate in a reasonable amount of time. The second is whether the agents’ language can be thought of as “natural language”, i.e., symbols are assigned to meanings that make intuitive sense in terms of our conceptualization of the world.
4 LEARNING TO COMMUNICATE
Our first question is whether agents converge to successful communication at all. We see that they do: agents almost perfectly coordinate in the 1k rounds following the 10k training games for every architecture and parameter choice (Table 1).
We see, though, some differences between different sender architectures. Figure 2 (left) shows performance on a sample of the test set as a function of the first 5,000 rounds of training. The agents
converge to coordination quite fast, but the informed sender reaches higher levels more quickly than the agnostic one.
The informed sender makes use of more symbols from the available vocabulary, while the agnostic sender constantly uses a compact 2-symbol vocabulary. This suggests that the informed sender is using more varied and word-like symbols (recall that the images depict 463 distinct objects, so we would expect a natural-language-endowed sender to use a wider array of symbols to discriminate among them). However, it could also be the case that the informed sender vocabulary simply contains higher redundancy/synonymy. To check this, we construct a (sampled) matrix where rows are game image pairs, columns are symbols, and entries represent how often that symbol is used for that pair. We then decompose the matrix through SVD. If the sender is indeed just using a strategy with few effective symbols but high synonymy, then we should expect a 1- or 2-dimensional decomposition. Figure 2 (right) plots the normalized spectrum of this matrix. While there is some redundancy in the matrix (thus potentially implying there is synonymy in the usage), the language still requires multiple dimensions to summarize (cross-validated SVD suggests 50 dimensions).
We now turn to investigating the semantic properties of the emergent communication protocol. Recall that the vocabulary that agents use is arbitrary and has no initial meaning. One way to understand its emerging semantics is by looking at the relationship between symbols and the sets of images they refer to.
The objects in our images were categorized into 20 broader categories (such as weapon and mammal) by McRae et al. (2005). If the agents converged to higher level semantic meanings for the symbols, we would expect that objects belonging to the same category would activate the same symbols, e.g., that, say, when the target images depict bayonets and guns, the sender would use the same symbol to refer to them, whereas cows and guns should not share a symbol.
To quantify this, we form clusters by grouping objects by the symbols that are most often activated when target images contain them. We then assess the quality of the resulting clusters by measuring their purity with respect to the McRae categories. Purity (Zhao & Karypis, 2003) is a standard measure of cluster “quality”. The purity of a clustering solution is the proportion of category labels in the clusters that agree with the respective cluster majority category. This number reaches 100% for perfect clustering and we always compare the observed purity to the score that would be obtained from a random permutation of symbol assignments to objects. Table 1 shows that purity, while far from perfect, is significantly above chance in all cases. We confirm moreover that the informed sender is producing symbols that are more semantically natural than those of the agnostic one.
Still, surprisingly, purity is significantly above chance even when the latter is only using two symbols. From our qualitative evaluations, in this case the agents converge to a (noisy) characterization of objects as “living-vs-non-living” which, intriguingly, has been recognized as the most basic one in the human semantic system (Caramazza & Shelton, 1998).
Rather than using hard clusters, we can also ask whether symbol usage reflects the semantics of the visual space. To do so we construct vector representations for each object (defined by its ImageNet label) by averaging the CNN fc representations of all category images in our data-set (see Section 3 above). Note that the fc layer, being near the top of a deep CNN, is expected to capture highlevel visual properties of objects (Zeiler & Fergus, 2014). Moreover, since we average across many specific images, our vectors should capture rather general, high-level properties of objects.
We map these average object vectors to 2 dimensions via t-SNE mapping (Van der Maaten & Hinton, 2008) and we color-code them by the majority symbol the sender used for images containing the corresponding object. Figure 3 (left) shows the results for the current experiment. We see that objects that are close in CNN space (thus, presumably, visually similar) are associated to the same symbol (same color). However, there still appears to be quite a bit of variation.
4.1 OBJECT-LEVEL REFERENCE
We established that our agents can solve the coordination problem, and we have at least tentative evidence that they do so by developing symbol meanings that align with our semantic intuition. We
turn now to a simple way to tweak the game setup in order to encourage the agents to further pursue high-level semantics.
The strategy is to remove some aspects of “common knowledge” from the game. Common knowledge, in game-theoretic parlance, are facts that everyone knows, everyone knows that everyone knows, and so on (Brandenburger et al., 2014). Coordination can only occur if the basis of the coordination is common knowledge (Rubinstein, 1989), therefore if we remove some facts from common knowledge, we will preclude our agents from coordinating on them. In our case, we want to remove facts pertaining to the details of the input images, thus forcing the agents to coordinate on more abstract properties. We can remove all low-level common knowledge by letting the agents play only using class-level properties of the objects. We achieve this by modifying the game to show the agents different pairs of images but maintaining the ImageNet class of both the target and distractor (e.g., if the target is dog, the sender is shown a picture of a Chihuahua and the receiver that of a Boston Terrier).
Table 2 reports results for various configurations. We see that the agents are still able to coordinate. Moreover, we observe a small increase in symbol usage purity, as expected since agents can now only coordinate on general properties of object classes, rather than on the specific properties of each image. This effect is clearer in Figure 3 (right), when we repeat t-SNE based visualization of the relationship that emerges between visual embeddings and the words used to refer to them in this new experiment.
5 GROUNDING AGENTS’ COMMUNICATION IN HUMAN LANGUAGE
The results in Section 4 show communication robustly arising in our game, and that we can change the environment to nudge agents to develop symbol meanings which are more closely related to the visual or class-based semantics of the images. Still, we would like agents to converge on a language fully understandable by humans, as our ultimate goal is to develop conversational machines. To do this, we will need to ground the communication.
Taking inspiration from AlphaGo (Silver et al., 2016), an AI that reached the Go master level by combining interactive learning in games of self-play with passive supervised learning from a large set of human games, we combine the usual referential game, in which agents interactively develop their communication protocol, with a supervised image labeling task, where the sender must learn to assign objects their conventional names. This way, the sender will naturally be encouraged to use such names with their conventional meaning to discriminate target images when playing the game, making communication more transparent to humans.
In this experiment, the sender switches, equiprobably, between game playing and a supervised image classification task using ImageNet classes. Note that the supervised objective does not aim at improving agents’ coordination performance. Instead, supervision provides them with basic grounding in natural language (in the form of image-label associations), while concurrent interactive game playing should teach them how to effectively use this grounding to communicate.
We use the informed sender, fc image representations and a vocabulary size of 100. Supervised training is based on 100 labels that are a subset of the object names in our data-set (see Section 3 above). When predicting object names, the sender uses the usual game-embedding layer coupled with a softmax layer of dimensionality 100 corresponding to the object names. Importantly, the game-embedding layers used in object classification and the reference game are shared. Conse-
quently, we hope that, when playing, the sender will produce symbols aligned with object names acquired in the supervised phase.
The supervised objective has no negative effect on communication success: the agents are still able to reach full coordination after 10k training trials (corresponding to 5k trials of reference game playing). The sender uses many more symbols after training than in any previous experiment (88) and symbol purity dramatically increases to 70% (the obs-chance purity difference also increases to 37%).
Even more importantly, many symbols have now become directly interpretable, thanks to their direct correspondence to labels. Considering the 632 image pairs where the target gold standard label corresponds to one of the labels that were used in the supervised phase, in 47% of these cases the sender produced exactly the symbol corresponding to the correct supervised label for the target image (chance: 1%).
For image pairs where the target image belongs to one of the directly supervised categories, it is not surprising that the sender adopted the “conventional” supervised label to signal the target . However, a very interesting effect of supervision is that it improves the interpretability of the code even when agents must communicate about images that do not contain objects in the supervised category set. This emerged in a follow-up experiment in which, during training, the sender was again exposed (with equal probability) to the same supervised classification task as above, but now the agents played the referential game on a different dataset of images derived from ReferItGame (Kazemzadeh et al., 2014). In its general format, the ReferItGame contains annotations of bounding boxes in real images with referring expressions produced by humans when playing the game. For our purposes, we constructed 10k pairs by randomly sampling two bounding boxes, to act as target and distractor. Again, the agents converged to perfect communication after 15k trials, and this time used all 100 available symbols in some trial.
We then asked whether this language was human-interpretable. For each symbol used by the trained sender, we randomly extracted 3 image pairs in which the sender picked that symbol and the receiver pointed at the right target (for two symbols, only 2 pairs matched these criteria, leading to a set of 298 image pairs). We annotated each pair with the word corresponding to the symbol in the supervised set. Out of the 298 pairs, only 25 (8%) included one of the 100 words among the corresponding referring expressions in ReferItGame. So, in the large majority of cases, the sender had been faced with a pair not (saliently) containing the categories used in the supervised phase of its training, and it had to produce a word that could, at best, only indirectly refer to what is depicted in the target image. We then tested whether this code would be understandable by humans. In essence, it is as if we replaced the trained agent receiver with a human.
We prepared a crowdsourced survey using the CrowdFlower platform. For each pair, human participants were shown the two images and the sender-emitted word (that is, the ImageNet label associated to the symbol produced by the sender; see examples in Figure 4). The participants were asked to pick the picture that they thought was most related to the word. We collected 10 ratings for each pair.
We found that in 68% of the cases the subjects were able to guess the right image. A logistic regression predicting subject image choice from ground-truth target images, with subjects and words as random effects, confirmed the highly significant correlation between the true and guessed images
(z = 16.75, p < 0.0001). Thus, while far from perfect, we find that supervised learning on a separate data set does provide some grounding for communication with humans, that generalizes beyond the conventional word denotations learned in the supervised phase.
Looking at the results qualitatively, we found that very often sender-subject communication succeeded when the sender established a sort of “metonymic” link between the words in its possession and the contents of an image. Figure 4 shows an example where the sender produced dolphin to refer to a picture showing a stretch of sea, and fence for a patch of land. Similar semantic shifts are a core characteristic of natural language (e.g., Pustejovsky, 1995), and thus subjects were, in many cases, able to successfully play the referential game with our sender (10/10 subjects guessed the dolphin target, and 8/10 the fence). This is very encouraging. Although the language developed in referential games will be initially very limited, if both agents and humans possess the sort of flexibility displayed in this last experiment, the noisy but shared common ground might suffice to establish basic communication.
6 DISCUSSION
Our results confirmed that fairly simple neural-network agents can learn to coordinate in a referential game in which they need to communicate about a large number of real pictures. They also suggest that the meanings agents come to assign to symbols in this setup capture general conceptual properties of the objects depicted in the image, rather than low-level visual properties. We also showed a path to grounding the communication in natural language by mixing the game with a supervised task.
In future work, encouraged by our preliminary experiments with object naming, we want to study how to ensure that the emergent communication stays close to human natural language. Predictive learning should be retained as an important building block of intelligent agents, focusing on teaching them structural properties of language (e.g., lexical choice, syntax or style). However, it is also important to learn the function-driven facets of language, such as how to hold a conversation, and interactive games are a potentially fruitful method to achieve this goal.

In this paper, a referential game is proposed between two agents. Both agents observe two images. The first agent, called the sender, receive a binary target variable (t) and must send a symbol (message) to the second agent, called the receiver, such that this agent can recover the target. The agents both get a reward, if the receiver agent can predict the target. The paper proposes to parametrize the agents as neural networks - with pretrained representations of the images as feature vectors - and train them using REINFORCE. In this setting, it is shown that the agents converge to  optimal policies and that their learned communications (e.g. the symbolic code transmitted from the sender to the receiver) have some meaningful concepts. In addition to this, the paper presents experiments on a variant of the game grounded on different image classes. In this setting, the agents appear to learn even more meaningful concepts. Finally, multi-game setup is proposed, where the sender agent is alternating between playing the game before and playing a supervised learning task (classifying images). Not surprisingly, when anchored to the supervised learning task, the symbolic communications have even more meaningful concepts.
Learning shared representations for communication in a multi-agent setup is an interesting research direction to explore. This is a much harder task compared to standard supervised learning or single-agent reinforcement learning tasks, which justifies starting with a relatively simple task. To the best of my knowledge, the approach of first learning communication between two agents and then grounding this communication in human language is novel. As the authors remark, this may be an alternative paradigm to standard sequence-to-sequence models which tend to focus on statistical properties of language rather than their functional aspects. I believe the contributions of the proposed task and framework, and the analysis and visualization of what the communicated tokens represent is a useful stepping stone for future work. For this reason, I think the paper should be accepted.
Other comments:
- How is the target (t) incorporated into the sender networks? Please clarify this.
- Table 1 and Table 2 use percentage (%) values differently. In the first, percentages seem to be written in the interval [0, 100], and in the second in the interval [0, 1]. Please correct this. Perhaps related to this, in Table 1, the column "obs-chance purity" seems to have extremely small values. I assume this was mistake?
- "assest" -> "assess"
- "usufal" -> "usual"