Human communication is a collaborative process. Speakers, on top of conveying their own intent, adjust the content and language expressions by taking the listeners into account, including their knowledge background, personalities, and physical capabilities. Towards building AI agents with similar abilities in language communication, we propose Pragmatic Rational Speaker (PRS), a framework extending Rational Speech Act (RSA). The PRS attempts to learn the speaker-listener disparity and adjust the speech accordingly, by adding a light-weighted disparity adjustment layer into working memory on top of speaker's long-term memory system. By fixing the long-term memory, the PRS only needs to update its working memory to learn and adapt to different types of listeners. To validate our framework, we create a dataset that simulates different types of speaker-listener disparities in the context of referential games. Our empirical results demonstrate that the PRS is able to shift its output towards the language that listeners are able to understand, significantly improve the collaborative task outcome. In human communication, speakers often adjust their language production by taking into consideration listeners' personality, background knowledge, perceptual or physical capabilities etc (Clark, 1996). Recent years have seen an increasing amount of work that explores pragmatic reasoning based on Rational Speech Act (RSA) (Andreas and Klein, 2016; Fried et al., 2018a,b; White et al., 2020; Cohn-Gordon et al., 2018), multi-agent emergent communication framework (Lazaridou et al., 2020; Lazaridou and Baroni, 2020), and Theory of Mind in communication (Bara et al., 2021; Zhu et al., 2021). However, except for (Zhu et al., 2021),  Work done during undergraduate study at the University of Michigan. most previous works assume that the listeners and the speakers have the same background knowledge and capabilities, including vocabulary size, visual access, and relative locations. This assumption is a great simplification of real-world communication where speakers and listeners often have various types of disparities. To address this limitation, this paper extends the Rational Speech Act (RSA) (Frank and Goodman, 2012) model towards rational agents learning to adapt behaviors based on their experience with the listener. The design choice of our model is inspired by the human cognitive system (Cowan, 2008; Wardlow, 2013) where a limited capacity working memory is built on top of the long-term memory to adjust the output to be task and environment specific. Each communication is a modification on the long-term memory (Reed, 2012) with situation-specific factors. In our framework, we fix the long-term memory which captures lan-2829 guage structure for communication, and introduce a light-weighted working memory (Miyake and Shah, 1999) for the Pragmatic Rational Speaker to modify and accommodate two goals: 1) a task goal which retrieves relevant information from the long-term memory and accomplish the task, and 2) a disparity goal which learns and adjusts the conversation to accommodate the listener's disparity through reinforcement learning. We separate each component as they are independent of each other in utility, and can be easily switched and adapted for new tasks and new environment. Different from previous works which only demonstrate how learned models affect task performance (e.g. (Shridhar et al., 2020; Zhu et al., 2021; Corona et al., 2019)), one of our goals is to also provide transparency on what models have indeed learned towards the end goal. It's well established that end-to-end neural models can often take advantage of spurious data bias to gain end performance. Models that only report end measure without showing their internal works would not be sufficient to tell the whole story about model's abilities. To serve this goal, we situated our investigation in the context of a referential game 1 as shown in Figure 1. We carefully curated a dataset to simulate two types of disparity: knowledge disparity and perceptual disparity . Our empirical results demonstrate that our model is able to significantly improve the collaborative game performance by shifting communication towards the language that the listeners with disparities are able to understand. In addition, our results show that separating working memory from long-term memory leads to faster learning and better performance than the previous model which conducted joint end-to-end learning. Our contributions are the following. 1) Following human cognition, we demonstrate the benefits of separating working memory from the long-term memory, compared to end-to-end joint training. 2) We propose a new dataset to simulate multiple distinct types of disparities, and demonstrate the pragmatic adaptability of our model. 3) Instead of focusing on mere end task performance, we show model's strong language shift ability to accommodate listener's disparities. 1 Different from traditional referential ground work as (Liu et al., 2013; Gorniak and Roy, 2004; Siebert and Schlangen, 2008; DeVault et al., 2005; Liu et al., 2012), we adopted this term from a recent line of work (Lazaridou et al., 2020; Andreas and Klein, 2016) to refer to the task described in Figure 1. The dataset and code are available through https://github.com/sled-group/ Pragmatic-Rational-Speaker to facilitate future work on pragmatics and theory of mind in language interpretation and generation. It has been studied (Leung et al., 2021; Stephens et al., 2010; Wardlow, 2013) in psychology that human speakers adjust the way how we speak for successful communication after learning the listener's disparity. Some recent work (Zarrie and Schlangen, 2019; Zhu et al., 2021; Corona et al., 2019; Hawkins et al., 2021) attempt to address similar questions. We build our model upon the following two concepts. The Rational Speech Act (RSA) model (Frank and Goodman, 2012) is a probabilistic model for the speakers and listeners to pragmatically reason about each other's intention. In the context of a referential game (Monroe and Potts, 2015), for example (Figure 1), given an image m , it starts with a literal speaker S 0 to generate caption c : PS 0 ( c | m ) . A rational listener L 1 reasons about the literal speaker's ( S 0 ) strategy and picks the best image that matches the description. A rational speaker S 1 then takes the rational listener's ( L 1 ) strategy into account and produces a caption c that maximizes the collaborative game goal. In previous work (Andreas and Klein, 2016) and (Lazaridou et al., 2020; Lazaridou and Baroni, 2020), the same referential game setup was used to propose a rational speaker that learns to reason the collaborative game and to produce natural sounding image captions based on RSA. However, they were mainly addressing the task goal , assuming the speaker and listener have the exact same capabilities and knowledge background, which is unrealistic. In our work, we created listeners with disparity d and extend this model for the speaker to accommodate both the task and disparities goals. Working memory (also short-term memory) is used in neuropsychology and cognitive science (Cowan, 2008; Miyake and Shah, 1999) to refer to the memory that controls attention, plans and carries out behavior. It is a combination of multiple components, including the contribution of long-term memory (Reed, 2012; Sawangjit et al., 2018) and situation-specific task processing (Funahashi, 2017). The classical artificial intelligence work such as ACT (Heise and Westermann, 1989) and SOAR (Laird et al., 1987) also incorporated the concept of working memory to model human short-term memory. The similar concept has been used in recent work such as (Hermann et al., 2017; Hill et al., 2017). Our work is a novel application of the working memory to pragmatically adjust communication for speaker-listener disparities ( disparity goal), and take advantage of the internal simulation architecture to achieve the task goal. Similar to (Kottur et al., 2017; Lazaridou et al., 2020), our model learns to converge language to adapt to listener's disparities through interactions, instead of ground truth supervision on language generation. The speakers have zero prior knowledge on the listener's background nor an oracle access to probe the listener's brain. Different from previous works, our model is able to generalize to distinct types of disparities. In addition, while previous models were trained in an end-to-end joint fashion, our work separates training and demonstrates the efficiency of working memory. Most importantly, few of the previous work were able to showcase model's language capabilities and only evaluate them by the end performance (e.g. accuracy), whereas our work emphasizes on evaluating how well the models learn to shift the language towards better understanding. There are many levels of disparities during verbal communication (Stephens et al., 2010), including phonetic, lexical, grammatical, semantic representations, etc. In our work, we assembled two datasets, and challenge the speaker model to handle two types of disparities: 1) knowledge disparity, and 2) perceptual disparity. The knowledge disparity is simulated through the hypernym dataset, where the listener only understands the hypernym for all the objects (e.g. food instead of pizza), whereas the speaker understands both. This dataset challenges the speaker model at the lexical level to learn what listener's vocab limitation, and shift towards the words that they understand. The perceptual disparity is simulated through the limited visual dataset, where the listener has impaired vision or some objects were physically blocked from the eyesight. This dataset challenges the speaker to shift attention and pick the visible objects for the listener to describe. For control and demonstration purposes, we remove all the animal-related objects and words from listener's training. These datasets are used to simulate listener's disparities and train the listener's model as described in Section 4.2. The speaker's long term memory was trained with the original data which has full knowledge of the vocab and objects, but no idea what the listeners are or aren't capable of. Detailed dataset components can be found in the Appendix. We modified the Abstract Scenes (Gilberto Mateos Ortiz et al., 2015) dataset for our experiments. There are 10020 images, each including 3 ground truth captions, and a median of 6 to 7 objects. We assembled  35k pairs of images that differ by  4 objects as the Hard set,  25k pairs that differ by > 4 objects as the Easy set, and together as the Combined set. The image pairs were split into training, validation and testing by a ratio of 8:1:1. Given a pair of images m 0 , m 1 , the target image indicator t  { 0 , 1 } , and the listener's disparity d , the speaker generates a caption c for the target image m t , and the listener needs to pick out the correct target t given c . Both receive a reward of +1 upon correct choice, and  1 otherwise. we start by building the Literal Speaker S 0 , gradually increase model structure and functionality with the vanilla Rational Speaker S 1 and the Pragmatic Rational Speaker S d1 . Upon retrieving a list of candidate captions C from the long-term memory, the final goal for S d1 is to output the best caption c in the working memory, that accommodates both 1) task goal: describes the unique features of the target image, and 2) disparity goal: learns and accommodates the listener's disparity. Table 1 is a brief summary of each model. The Literal Speaker S 0 generates candidate captions c for a given image m (Eq 1), which serves as the long-term memory. The Rational Listener L 1 picks out an image as the target given speaker's description (Eq 2). The vanilla Rational Speaker S 1 achieves the task goal by simulating the listener's mind internally in its working memory (Eq 3). L d 1 incorporates disparity to the Rational Listener. The Pragmatic Rational Speaker S d1 adds a light-weight disparity adjustment layer (Eq 5) to learn and accommodate listener's disparity through interactions, and achieves both goals. Each component can be easily switched and adapted to new tasks or environment. L d 1 : P ( t | m 0 , m 1 , c, d )  PS 1 ( c | m 0 , m 1 , t, d )  P ( t | m 0 , m 1 , d ) (4) S d1 : P ( c | m 0 , m 1 , t, d )  PL d 1 ( t | m 0 , m 1 , c, d )  P ( c | m 0 , m 1 , d ) (5) The Literal Speaker S 0 (Figure 2) is an object detection based image captioning module that generates caption candidates for the target image. o 1 , . . . , o k , b 1 , . . . , b k = ObjDet ( m t ) e 1 , . . . , e k = WordEmb ( o 1 , . . . , o k ) c 1 , . . . , c n = Transformer ( e 1 , . . . , b 1 , . . . ) (6) For a given target image m t , since it's important to ground words to the scenes in order to control the disparities in vocabularies, we applied the object detector YOLO3 (Redmon and Farhadi, 2018) to extract a list of k detected objects O = { o 1 , o 2 , . . . , o k } , and their corresponding bounding boxes B = { b 1 , b 2 , . . . , b k } . Each image chooses at most max _ obj = 9 detected objects, and the names of each were embedded with a pre-trained BERT (Devlin et al., 2019) word embedding E = { e 1 , e 2 , . . . , e k } . These embed-dings are then concatenated with their bounding box locations, and sent to the Transformer Decoder to generate beam _ size = 30 candidate captions C = { c 1 , c 2 , . . . , c n } for each target image. Without disparity concerns, the Rational Listener picks out the image that they believe is the target. g 0 = FT _ Transformer ( m 0 , c ) g 1 = FT _ Transformer ( m 1 , c ) t = argmax i { 0 , 1 } CosSim ( g i , c ) (7) Recall that S 0 used a Transformer decoder to connect the image and its corresponding captions. We reuse the same Fixed pre-trained Training-mode Transformer module (named FT _ Transformer ) to decide which image does the caption ground better in. Adopting the idea of teacher-forcing language training, the output ( g i ) of FT _ Transformer with an input pair ( m i , c ) should closely resemble the original input c if the input image m i is indeed the one used to generate the caption c . By calculating the co-sine similarity of each ( g i , c ) pair, the image that grounds better (higher CosSim ) in the description would be chosen as the target. This module allows the agents to quickly and accurately make the decisions without further training. In theory, if the speaker and the listener were to have the exact same brain (same model and weights), the performance of this task should approach 100%. The results of No Disparity speaker in Figure 3 confirmed the design choice. Without disparity concerns, the Rational Speaker ( S 1 ) fulfills the task goal by simulating (Figure 2) the Rational Listener ( L 1 )'s behavior, and rank the candidate captions generated by the Literal Speaker ( S 0 ) according to how well they can describe the target image apart from the distractors. This design is under the fair assumption that both speakers and listeners are aware of the collaborative game goal, but can be switched for other task purposes. For i  { 0 ,    , n } , where n = | C | : t i , p i = Simulate _ L 1 ( m 0 , m 1 , c i ) c = c argmax i [[ t i == t  ]]  p i (8) Given an image pair ( m 0 , m 1 ), and a list of candidate captions C = { c 1 ,    , c n } generated by S 0 , the Rational Speaker goes through each caption c i and simulates how well the listener ( Simulate _ L 1 ) would pick out the correct target image. If a candidate caption c i helps the simulator pick out the correct target image (i.e. t i == t  ) with high confidence ( p i ), then it will be chosen as the final caption sent over to the actual listener. The simulated listener shares the same architecture as L 1 and initializes the weights pre-trained from S 0 . By doing so, the Rational Speaker takes the listener's intention into account and achieves the task goal. In the real world, however, it is hardly the case that different agents have the exact same knowledge background, experiences, physical capabilities, etc. The listener's decision making process is influenced by various kinds of disparities d . To study speaker's ability of situated language adjustment, we created two representative types of listeners with different knowledge background and visual capabilities by training different caption grounding modules ( FT _ Transformer ) with the datasets assembled in Section 3. These disparities would challenge the speaker model to adjust the language at different levels. 1. L d 11 : Hypernym. With limited vocabulary and knowledge in a certain domain, people tend to refer to objects in their hypernym form (e.g. animal instead of cat). In this experiment, we create listeners that would refer to all the detected objects by their hypernyms. This disparity would require the speaker to switch individual words that share similar meanings. 2. L d 21 : Limited Visual. Due to the physical orientation or impaired vision capability, it is likely that some objects are blocked or hardly visible to one party but not the other. In this experiment, we remove all the animal objects from listener's visual detected object list ( O ), and replace the relevant descriptions with the special token [UNK]'. This disparity would require the speaker to shift attention, and choose alternative objects to describe. We investigate in listeners with a subset of speaker's capabilities under the argument that in the opposite case, the listener could use only a subset of the knowledge to achieve best performance without having the speakers to adjust the speech. Other disparities can be inferred through transfer learning or are left for further investigation with broader information access and datasets. On top of the Rational Speaker ( S 1 ), the Pragmatic Rational Speaker incorporates a disparity adjustment layer to learn and accommodate the listener's disparity through emergent communication. For i  { 0 ,    , n } , where n = | C | : q i = MLP ( SentenceEmb ( c i )) a i = [[ t i == t  ]]  p i  q i c = c argmax i a i (9) We use a pretrained BERT model to embed each candidate caption c i , add a single MLP layer, and approximate the REINFORCE policy through Equation 9. The reward ( r c  ) for each chosen caption c  is +1 or  1 . The loss is calculated for all the chosen captions across each batch (Eq 10). We conducted the same sets of experiments using individual words (object names) instead of sentences to demonstrate the effects of working memory on disparity accommodation and internal task simulation, reducing the noise that came from the imperfection of the image description generator. The simplified pipeline uses the detected object name embedding for disparity adjustment, and the listener picks the target images by conducting simple word matching. and Balance of Goals. Recall that each speaker model has different capabilities (Table 1) and only S d 1 is able to fulfill both task and disparity goals. Implementation details and more experiment results can be found in Appendix. 1. [Task Performance] that measures overall accuracy of the collaborative game. Task performance is often the sole evaluation metrics in previous work. 2. [Efficiency] that measures time used for model training across tasks. 3. [Transparency] that uncovers the underlying distribution shift of vocabulary use learned to accommodate different types of disparities. 4. [Balance of Goals] that the working memory needs to consider between the task and disparity goals to achieve maximum performance 5.1 Task Performance Comparison To assess the performance of the speakers in the collaborative game, Figure 3 presents the task accuracies with Literal Speaker ( S 0 ), Rational Speaker ( S 1 ), Pragmatic Rational Speaker ( S d 1 ), and No Disparity ( S nd 1 ). S nd 1 has the same structure as S 1 and was trained on the same disparity dataset as the corresponding listener. It serves as the upper bound of performance. The same experiments also were conducted at the word level. For each type of listener disparity, the performance is S 0 << S 1 < S d 1 < S nd 1 . The vanilla Rational Speaker ( S 1 ) improved the overall performance from Literal Speaker by over 25% because it is achieving the task goal to describe the target 2834 image apart from the distractor. The Pragmatic Rational Speaker ( S d 1 ) is able to learn and adjust for the listener's disparity, and further improve the game performance by  10%. There is still, however a gap between S d 1 and the upper bound S nd 1 , where the speaker and the listener have the exact knowledge and capability limitation, potentially due to the imperfection in caption generations. Breaking down between the hard , easy datasets in Figure 4 (recall that image pairs that differ by  4 objects are in the Hard set, otherwise the Easy set), S d 1 on the easy dataset is able to gain a lot more improvement upon its Rational Speaker compared to the pair trained on the hard dataset. The gap between S d 1 and No Disparity is also a lot smaller for the model trained on the easy dataset. This is likely because when a pair of images differ more objects (easier), the model has more options to adjust upon, hence the larger improvement. Compared to the sentence level model, the word level pragmatic speaker for L d 11 achieves even higher improvement against the corresponding Rational Speaker. They both achieve almost perfect accuracy with close to zero gap to the upper bound. This suggests the high potential of the disparity adjustment design, especially after reducing the caption generation and interpretation noise. To study the training efficiency of the working memory, we compared our model to the joint training Multi-Task leaning model in (Lazaridou et al., 2020)'s work, retrained and evaluated in our dataset. The image captioning model and the REINFORCE Train(min) Accuracy% BLEU4 Joint 19.04 60.14 27.79 Separate 21.02 77.34 29.3  LM 11.59 29.3  WM 9.43 77.34 Functional in our task refers to the REINFORCE learning to achieve both task and disparity goals (evaluated by Accuracy), and structural refers to the caption generation loss for natural-sounding language (evaluated by BLEU4). We used  f =  s = 1 as in previous work for our experiments. Detailed training and comparison strategies can be found in the Appendix. Table 2 shows that for each type of disparity, our model separating working memory from long-term memory is able to achieve higher accuracy and higher BLEU4 score than the joint training. Moreover, the Joint Trained model needs to retrain all the weights for each type of disparity from scratch, whereas our model only needs to train the long-term memory once, and retrain the light weighted working memory for each type of disparity, which is much more efficient. To gain insights in whether the Pragmatic Rational Speaker (PRS) is actually adjusting the descriptions for listeners' disparities or taking the advantage of statistical bias to achieve higher task performance, we plotted the word distribution shift across different types of disparities. Qualitative examples can be found in Figure 6. For each experiment, the word frequencies of all the chosen captions were calculated for the Rational Speakers, the Pragmatic Rational Speakers, and Joint Training. We collected the top choice of each speaker per image 2835 the mean and standard deviation in Figure 5. In the Hypernym disparity (Figure 5a) experiment, where the listener only understands the hypernym of detected objects, the lower-case words on the left are the top detected object names, and the upper-case words on the right are hypernyms. On the left side, the word frequencies of PRS significantly dropped from the Rational Speaker. On the right side, the model is maintaining similar level, or using some of the hypernyms more frequently (y-axis in log scale). Note that the Rational Speaker can generate both hypernym and hyponym regardless of disparities, and multiple valid captions available for all speakers to choose from. For the Joint Trained Speaker, we also observed a hyponym usage drop (left), but it's unclear how it accommodates the disparity without using hypernyms. This result shows that PRS learned to avoid using hyponyms, and replaced them with their hypernym to accommodate the disparity. For the Limited Visual disparity (Figure 5b), since all the animal objects are missing for the listener, there is a sharp decline in S d 21 's use of animal related words during the communication. Instead, it is choosing other objects such as hat, and ball to describe the target image. The PRS is accommodating listener L d 21 's disparity by shifting the attention and choosing alternative objects other than animals to communicate. The behavior of the Joint Trained Speaker is harder to interpret. Recall that the working memory of the Pragmatic Rational Speaker ( S d 1 ) has two two goals: 1) Task Goal : an internal simulation of a listener to rank the candidate captions by their uniqueness in describing the target image, and 2) Disparity Goal : a disparity adjustment layer to learn and accommodate the listener's disparity through interactions. Each goal component can be formalized in the above two terms (Equation 11). We parameterized each term with  l and  d to study how different  l :  d weight ratio could affect rational speaker's ability to achieve both goals. Figure 7 shows that when the Pragmatic Rational Speaker puts a high emphasis on adjusting the listener's disparity  d , it would forget to describe the unique characters of the target image and lower the overall performance. On the other hand when the PRS emphasize too much on the task goal, it would forget to accommodate listener's disparities, and lower the overall performance as well. In the end, we chose  l :  d = 1 : 1 for all experiments demonstrated above. In this work, we present a novel framework based on the Rational Speech Act framework for pragmatic communication that can adjust the conversation content according to listener's disparities by adding a light-weighted working memory on top of speaker's long-term memory. The Pragmatic Rational Speaker significantly improved the collaborative game performance by shifting the conversation towards the language that the listeners are able to understand. There are, however, several limitations that requires further investigation. First of all, despite recent progress, algorithms that connect language and the visual world are still limited. For example, caption generation, even in this simple setup, often does not faithfully capture what's been conveyed in the images. As our framework heavily relies on the quality of various models that bridge language and vision, e.g., as part of our long term memory, it's important to improve functionality and performance of these base models. We conducted our experiments in a relative simple and artificial environment with the purpose of easy control and demonstration. We emphasize on evaluating model's actual language ability of adjusting for the disparities on top of task performance. The next step would be to apply the framework to more realistic images and interactive environment. Other than listener's knowledge background and perceptual capabilities, there are a lot of other reasons for language communication to be adjusted, such as the physical environment, relative positions, speaker's personalities, etc. Studying how a rational agent can accommodate these disparities would require additional multimodal datasets and information processing methods. At the moment, the Pragmatic Rational Speaker trains a new layer in working memory from scratch for each type of disparity. This could have backward influence on the long-term memory. In lifelong learning (Parisi et al., 2019) like humans, the working memory can shape their long-term memory. At the very least, the model could store each learned disparity adjustments for future encounter. This modification is left for future work. Last but not least, instead of training for every single type of disparity to name, human learners have the ability of meta-learning and zero-shot transferring existing knowledge to a new category. Future work on pragmatic reasoning should be easily adaptable to different disparities and situations. This work was supported in part by the National Science Foundation under grant IIS-1949634. The authors would like to thank the anonymous reviewers for their valuable comments and suggestions.
abstain abstain objective abstain abstain method objective abstain other abstain abstain abstain abstain method abstain other objective abstain abstain abstain objective objective result objective objective objective result other method other method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain other abstain method abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain method abstain abstain method abstain abstain abstain method abstain method abstain abstain method abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain method result abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain objective abstain abstain abstain abstain result objective objective abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other other