summaries: The paper examines the effect of reward shaping in the context of emergent communication, and specifically with respect to the Shannon entropy of the emerged language. The paper showed that shaping the rewards can lead to higher entropy than training with just the base reward. We care about the entropy because this relates to how expressive the language is. There are existing works, such as Kharatinov et al 2020, which examine the relationship between task/environment and the resulting language entropy.  The paper uses PPO as its main outer-level optimizer, which it uses because the task is a multi-step task which might be challenging to learn with other algorithms such as REINFORCE.  The paper provides theoretical analysis of the effect of reward shaping on entropy by considering an analogy between PPO learning and a process they call the 'Extended Chinese Restaurant Process' (ECRP), where at each step, the customer can be divided into beta parts, and each of the beta bits of customers (which are now each of size 1/beta) takes a seat. This is presented as being analogous to how PPO first store multiple episodes over a buffer, and then updates the weights of the network, using the results of the whole buffer; then rinses and repeats.  Experimental results are presented which show that making the task harder/longer (by increasing the world radius) results in a fall of the entropy of the resultant emergent language, but not in the presence of reward shaping.  The paper asserts that whilst multi-step tasks, and PPO, are not much used in emergent communication literature currently, but it seems plausible that this will become the case in the future, and this paper is a small exploratory step in that direction. The paper looks into finding effects of reward shaping on emergent language learning with RL. The work shows the difference via analyzing entropy and the behavior of the learned agent. It draws similarities between the Chinese restaurant process and RL. And shows that some of the differences between the behavior of the agent learned with and without reward shaping can be explained by experience buffer size used in the RL learning algorithm. The work uses a new simple navigation task for the study. This paper investigates emergent language research, whose goal is to study language as it emerges from the inherent properties of the environment, language, and agents. This is typically in the context of reinforcement learning such that the phenomena arises from the pressure to maximize reward. This paper considers two types of rewards. One directly comes from the task. The other is shaped rewards which makes learning easier. This paper mainly investigates the impact of shaped reward on the emergent language. Using a simple sender-receiver navigation game, the paper shows that shaped rewards can explicitly bias the semantics of the learned language, change the entropy of the learned language, and mask the potential effects of other environmental variables of interest. This paper studies the effect of auxiliary rewards in an environment for a multi-agent communication task. The task is for a sender to navigate a receiver to the center of some environment, in worlds with varying sizes, and must develop a discrete language (consisting of 1 of 64 symbols) to solve this task.  The authors argue that seemingly innocuous ("unbiased") shaped rewards in the environment have significant, measurable effects on the learned languages, both qualitatively and quantitatively. The authors both visualize the effect of shaped rewards on the emergent language, and measure the change in entropy to the languages caused by the denser shaped rewards. For example, adding denser rewards increases the entropy (and presumably information) communicated by the language; denser rewards prevent a decrease in entropy for languages generated for larger (more difficult) tasks, suggesting that information bandwidth remains high in the shaped reward setting.  Finally, the authors suggest an alternative hypothesis for the differential entropies observed by language in this game: the size of the experience replay buffer in their PPO algorithm, which roughly corresponds to the frequency of updating the semantics of the learned language. There is some connection drawn here to a batched CRP with varying batch size, though it's not super clear to me how strong this connection is to PPO learning and why this CRP is the right way of thinking about experience buffer sizes.  The core argument of the paper, then, is to more carefully consider how environmental rewards might change the semantics of the learned language and mask other factors of interest in studies of emergent communication.
abstractText: One of the primary characteristics of emergent phenomena is that they are determined by the basic properties of the system whence they emerge as opposed to explicitly designed constraints. Reinforcement learning is often used to elicit such phenomena which specifically arise from the pressure to maximize reward. We distinguish two types of rewards. The first is the base reward which is motivated directly by the task being solved. The second is shaped rewards which are designed specifically to make the task easier to learn by introducing biases in the learning process. The inductive bias which shaped rewards introduce is problematic for emergent language experimentation because it biases the object of study: the emergent language. The fact that shaped rewards are intentionally designed conflicts with the basic premise of emergent phenomena arising from basic principles. In this paper, we use a simple sender-receiver navigation game to demonstrate how shaped rewards can 1) explicitly bias the semantics of the learned language, 2) significantly change the entropy of the learned language, and 3) mask the potential effects of other environmental variables of interest.
references.authors: David J. Aldous
references.title: Exchangeability and related topics
references.venue: École d’Été de Probabilités de Saint-Flour XIII —
references.authors: David Blei
references.title: The chinese restaurant process, 2007. URL https://www.cs.princeton.edu/ courses/archive/fall07/cos597C/scribe/20070921.pdf
references.authors: Nicolo’ Brandizzi Davide Grossi Luca Iocchi
references.title: Rlupus: Cooperation through emergent communication in the werewolf social deduction game, 2021
references.authors: Kalesha Bullard Douwe Kiela Franziska Meier Joelle Pineau Jakob Foerster
references.title: Quasiequivalence discovery for zero-shot emergent communication, 2021
references.authors: Rahma Chaabouni Eugene Kharitonov Emmanuel Dupoux Marco Baroni
references.title: Communicating artificial neural networks develop efficient color-naming systems
references.venue: Proceedings of the National Academy of Sciences,
references.authors: Tom Eccles Yoram Bachrach Guy Lever Angeliki Lazaridou Thore Graepel
references.title: Biases for Emergent Communication in Multi-agent Reinforcement Learning
references.venue: In Advances in Neural Information Processing Systems,
references.authors: Laura Harding Graesser Kyunghyun Cho Douwe Kiela
references.title: Emergent linguistic phenomena in multi-agent communication games
references.venue: In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP),
references.authors: Serhii Havrylov Ivan Titov
references.title: Emergence of language with multi-agent games: Learning to communicate with sequences of symbols
references.venue: Advances in Neural Information Processing Systems,
references.authors: Rishi Hazra Sonu Dixit Sayambhu Sen
references.title: Zero-shot generalization using intrinsically motivated compositional emergent protocols
references.venue: Visually Grounded Interaction and Language Workshop,
references.authors: Eric Jang Shixian Gu Ben Poole
references.title: Categorical reparameterization with gumbel-softmax
references.venue: In Proceedings of the 2017 International Conference on Learning Representations (ICLR),
references.authors: Natasha Jaques Angeliki Lazaridou Edward Hughes Caglar Gulcehre Pedro Ortega Dj Strouse Joel Z. Leibo Nando De Freitas
references.title: Social influence as intrinsic motivation for multi-agent deep reinforcement learning
references.venue: In International Conference on Machine Learning,
references.authors: Ivana Kajić Eser Aygün Doina Precup
references.title: Learning to cooperate: Emergent communication in multi-agent navigation
references.venue: In 42nd Annual Meeting of the Cognitive Science Society,
references.authors: Eugene Kharitonov Rahma Chaabouni Diane Bouchacourt Marco Baroni
references.title: Entropy minimization in emergent languages
references.venue: Proceedings of the 37th International Conference on Machine Learning,
references.authors: Satwik Kottur José Moura Stefan Lee Dhruv Batra
references.title: Natural language does not emerge “naturally” in multi-agent dialog
references.venue: Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing,
references.authors: Angeliki Lazaridou Karl Moritz Hermann Karl Tuyls Stephen Clark
references.title: Emergence of linguistic communication from referential games with symbolic and pixel input
references.venue: In International Conference on Learning Representations,
references.authors: Chris J. Maddison Andriy Mnih Yee Whye Teh
references.title: The concrete distribution: A continuous
references.authors: Schulman Filip Wolski Prafulla Dhariwal Alec Radford Oleg Klimov
references.title: Proximal policy
references.venue: mann. Stable baselines3
references.title: Reward Shaping, pp. 863–865
references.authors: Lazaridou
references.title: An image-based signalling game using rendered images from MuJoCo. The information being communicated is the shape and color of an object where set of shapes and colors are both small. Since the number of words available is at least as big as the cardinalities of the sets and the sender
sections.heading: 1 INTRODUCTION
sections.text: In emergent language research, the goal is to study language as it emerges from the inherent properties of the environment, language, and agents. One pitfall for such experiments, though, is that the language simply mirrors some of design choices of the environment or experimental setting more generally. there is a risk that the language simply mirrors the design choices of the environment. For example, Bullard et al. (2021) introduce a method for discovering optimal languages for communication between independently trained agents, yet rather than emerging from basic principles, the learned language is the result of an intentionally designed search algorithm. Reinforcement learning is a common tool in this field for observing the emergence of language out of a reward maximization pressure. One such design choice which can obscure these emergent properties is adding shaped rewards on top of the base reward of the reinforcement learning environment (Wiewiora, 2010).
  The base reward of the environment derives directly from succeeding at the task in question. The difficulty with relying solely on the base rewards is that if the task is especially long or complicated, the agent may only receive a reward infrequently which makes for a difficult learning problem. In such a case, base reward is considered sparse. This motivates shaped rewards which are inserted at intermediate steps based on domain knowledge in order to introduce an inductive bias towards good solutions. For example, the base reward in chess would simply be winning or losing the game. A shaped reward could then be taking the opponents material while not losing your own. While this shaped reward is often a good heuristic, it can lead to local optima; for example, it discourages strategies which would sacrifice individual pieces in order to win the whole game. While local optima present a problem for maximizing reward, the biases introduced by shaped rewards present a unique problem for emergent language which we will highlight in this paper.
  For emergent language research, the inductive bias which shaped rewards introduce is especially problematic because it exerts a significant influence on the learned language whose emergent properties are the object of study. This influence can comprise 1) biasing the semantics of the language, 2) changing a property of the whole language (e.g., language entropy), or 3) masking the influence of some other environmental parameter on the language. For example, some emergent language work incorporates shaped rewards into their environment without accounting for the the biases it may introduce (Mordatch & Abbeel, 2018; Brandizzi et al., 2021; Harding Graesser et al., 2019). From an engineering and design perspective, tweaking the system to achieve a desired result is standard
  practice, but from a scientific and experimental perspective, these additional shaped rewards serve as potential confounding factors and hinder accurate observations of the emergent phenomena.
  We study this by introducing a simple navigation game with continuous states and actions where a sender passes a one-word message to the receiver (illustrated in Figure 1b and 1a). Within this environment, our experiments look at the entropy and semantics of the emergent language in the absence and presence of shaped rewards. In the course of these experiments, we find that reinforcement learning algorithm’s experience buffer size has a significant impact on the entropy of the learned language and potentially explains our experimental findings. To this end we introduce a mathematical model based on the Chinese restaurant process for understanding the effect of experience buffer size on emergent language more generally. We highlight the following contributions in our paper:
  • Demonstrating basic ways in which shaped rewards can undesirably influence emergent language experiments
  • Presenting a mathematical model for understanding the role of experience buffer size in the entropy of emergent language
sections.heading: 2 RELATED WORK
sections.heading: 2.1 TAXONOMY
sections.text: We intentionally design our environment to study shaped rewards and the entropy of language, which requires it to differ from prior art in specific ways. To elucidate this, we create a taxonomy of emergent language environments based on whether or not the environment has multi-step episodes and the presence of a trivially optimal language (defined below). The taxonomy is given in Table 1 and a brief description of each environment is given in Appendix A.
  Generally speaking, the motivation for shaped rewards in a given environment is sparsity of the base reward which requires a multi-step, multi-utterance environment. Thus, our experiments naturally require a multi-step environment.
  We consider an environment to have a trivially optimal language if the information which needs to be communicated from sender to receiver can be perfectly represented in the emergent language. Such a language most frequently arises when the communicated information is derived from a small number of categorical variables encoded in the observation. For example, in an environment where the sender must specify an element of the set {red, green, blue}× {square, circle} using messages from the set {r, g, b}×{s, c}, a trivial language is where color maps to a unique letter in the first position and shape maps to a unique letter in the second position. Other environments have no trivially optimal languages. For example, if the sender must communicate an element of {1, 2, ..., 100} using messages from the set {a, b, c}, there is no trivially optimally language since the sender can at best partition the set of integers to reach an optimal but imperfect solution.
  Kharitonov et al. (2020) gives evidence that there is an entropy minimization in pressure inherent in emergent language settings. Building on this, Chaabouni et al. (2021) explicitly look at a the
  tradeoff between entropy and reward—higher entropy languages have the potential to convey more information at the expense of being more difficult to learn. This tradeoff disappears if a trivially optimal language is learned since there is no further reward maximization pressure. Language entropy greater than this minimum, then, does not emerge from the the reward maximization pressure. Although such an environment does not preclude studying entropy we choose to use an environment where the information being communicated is fully continuous so an increase in entropy can always translate to a increase in reward. This leads to a smooth tradeoff between entropy and reward which is illustrated in Figure 2.
sections.heading: 2.2 SPECIFIC APPROACHES
sections.text: The environment and agent configuration of this paper are most closely related to Chaabouni et al. (2021) who test the balance between entropy and semantic precision in a two-agent color discrimination task. Although the color space comprises 330 distinct colors, the environment facilitates languages which cover a (learned) region of nearby colors. In this way, there is no trivially optimal, one-to-one language as the task is inherently fuzzy. Havrylov & Titov (2017) use a signalling game with natural images which lacks a trivially optimal language, but using natural images results in many uncontrolled variables, lessening the ability of the experiments to make basic, first-principles claims.
  Studying shaped rewards has been explored previously in emergent language primarily as inductive bias for encouraging some property, such as compositionality, to emerge (Hazra et al., 2021; Jaques et al., 2019; Eccles et al., 2019). This paper, instead, focuses on the negative aspects of inductive biases introduced by shaped rewards regarding how they can hinder empirical investigation of emergent languages.
  Superficially, the environment used in this paper bears resemblance to Mordatch & Abbeel (2018) and Kajić et al. (2020) as these both deal with navigation. In both of these environments, the agents communicate about a discrete set of actions or goal locations, whereas the agents in this paper communicate about a continuous action. Neither of the papers specifically investigates the effects of shaped rewards on the languages learned.
sections.heading: 3 METHODS
sections.heading: 3.1 ENVIRONMENT
sections.text: In this paper we use a simple 2-dimensional navigation environment with two closely related tasks. A sender agent observes the position of a receiver agent, sends a message to the receiver, and the receiver takes an action. In the centerward task (illustrated in Figure 1a), the receiver is initialized uniformly at random within a circle and must navigate towards a circular goal region at the center. In the edgeward task, the receiver is initialized uniformly at random within a circle and must navigate to a goal region comprising the entire area outside of the circle. The centerward environment is a more realistic environment. The edgeward environment, on the other hand, admits a greater variety of languages, since moving consistently in any one direction eventually gets you to the edge; therefore, learning to move in a variety of evenly spaced direction solves the task more quickly but is not strictly necessary. There are no obstacles or walls in the environment. The receiver’s location and action are continuous variables stored as floating-point values. If the receiver does not reach the goal region within a certain number of steps, the episode ends with no reward given.
sections.heading: 3.2 AGENT ARCHITECTURE
sections.text: Our architecture comprises two agents, conceptually speaking, but in practice, they are a single neural network. The sender is a disembodied agent which observes the location of the receiver and passes a message in order to guide it towards the goal. The receiver is an agent which receives the message as its only input and takes an action solely based on that message (i.e., it is “blind”). The sender and receiver are randomly initialized at the start of training, trained together, and tested together. The architecture of the agents is illustrated in Figure 1b.
  The observation of the sender is a pair of floating-point values representing the receiver’s location. The sender itself is a 2-layer perceptron with tanh activations. The output of the second layer is passed to a Gumbel-Softmax bottleneck layer (Maddison et al., 2017; Jang et al., 2017) which enables learning a discrete, one-hot representation, as an information bottleneck (Tishby et al., 2000). The activations of this layer can be thought of as the “words” or “lexicon” of the emergent language. At evaluation time, the bottleneck layer functions deterministically as an argmax layer, emitting onehot vectors. The receiver is a 1-layer perceptron which receives the output of the Gumbel-Softmax layer as input. The output is a pair of floating-point values which determine the action of the agent. The action is clamped to a maximum step size.
sections.heading: 3.3 OPTIMIZATION
sections.text: Optimization is performed using stochastic gradient descent as a part of proximal policy optimization (PPO) (Schulman et al., 2017). Specifically, we use the implementation of PPO provided by Stable Baselines 3 with our neural networks implemented in PyTorch (Raffin et al., 2019; Paszke et al., 2019). Using a Gumbel-Softmax bottleneck layer allows for end-to-end backpropagation, making optimization faster and more consistent than using a backpropagation-free method like REINFORCE (Kharitonov et al., 2020; Williams, 1992).
  We will give a basic explanation of how PPO (and related algorithms) works as it is necessary to connect it to the mathematical model presented in Section 4. PPO has two stages sampling and optimizing. During the sampling stage, the algorithm runs the agents in the environment and stores the states, actions, and rewards in a experience buffer (or rollout buffer). The optimization stage then comprises performing gradient descent on the agents (i.e., neural network policy) using the data from experience buffer. The next iteration starts with the updated agents and an empty experience buffer.
sections.heading: 3.4 REWARDS
sections.text: We make use of two different rewards in our configuration, a base reward and an shaped reward. The base reward is simply a positive reward of 1 given if the receiver reaches to the goal region before the episode ends and no reward otherwise. The shaped reward, given at every timestep, is the decrease in distance to the goal. If the goal region is centered at (0, 0), the standard shaped reward for the centerward environment is given by Equation 1; we also use a trivially biased version of the
  reward which only takes into account horizontal distance specified in Equation 2. rt = √ x2t−1 + y 2 t−1 − √ x2t + y 2 t (1)
  r′t = √ x2t−1 − √ x2t (2)
  For the edgeward environment, we use the opposite of rt as the goal is to move away from the center.
  The interplay between base and shaped rewards is important to understand in the larger context of how reinforcement learning problems are structured. The base rewards are well-motivated and directly correspond to the ultimate aim of the task, but their sparsity can make it difficult for the agents to learn to succeed. Shaped rewards facilitate learning by using expert knowledge to form an inductive bias, yet they present a drawback for traditional reinforcement learning and emergent language. Within reinforcement learning where the goal is to train the best performing agent, shaped rewards can lead to the agent finding local optima if better solutions are excluded by the inductive bias. Within emergent language, the problem is more nuanced as the goal is primarily to study a wide range of emergent properties of the language learned within the environment. While base rewards have a natural connection to the environment and task, the shaped rewards introduce a reward signal which is not intrinsically connected with the task and environment, even if it is a good heuristic.
sections.heading: 4 EXPLANATORY MODEL
sections.text: We argue that the results presented in Sections 5.2 and 5.3 can be explained by an effective change in the size of PPO’s experience buffer. In order illustrate this, we introduce a simple mathematical model based on the Chinese restaurant process (Blei, 2007; Aldous, 1985). While the model does not exactly match our experimental setup, the key similarities allow us to reason about our results as well as potential future experiments.
  The Chinese restaurant process is an iterative stochastic process which yields a probability distribution over the positive integers. The key similarity between the Chinese restaurant process and our learning setup is that it they are self-reinforcing processes, that is, when a given value is selected in one iteration of the process, the probability that that value is chosen again in subsequent iterations increases. We generalize one aspect of the Chinese restaurant process in Section 4.2 to better match the sampling-optimization procedure of PPO. The primary simplification which this model makes is that it does not take into account the “meaning” of actions and the effects they have within the environment. For example, every successful agent in the centerward environment will use at least three distinct nearly equiprobably so as to span the 2-dimensional space whereas as no such lower bound exists in the stochastic process.
sections.heading: 4.1 CHINESE RESTAURANT PROCESS
sections.text: As the name suggests, a useful analogy for the Chinese restaurant process starts with a restaurant with infinitely many empty tables, indexed by the positive integers, which can hold an unbounded number of customers. As each customer walks in, they sit at a populated table with a probability proportional to the number of people already at that table. The customer will sit at a new table proportional to a hyperparameter α which modulates the concentration of the final distribution. The decision the customer makes is equivalent to sampling from a categorical distribution where the unnormalized weights are the customer counts along with the weight, α, for the new table. The pseudocode for the Chinese restaurant process is given in Algorithm 1 for β = 1. By analogy to the neural networks representing our agents, we can view the tables as bottleneck units and the customers choosing a table as parameter updates which reinforce the use of that unit in accordance with the reward. Mordatch & Abbeel (2018) implicitly assume this when they introduce a reward corresponding to the probability that the emergent lexicon is generated by a Chinese restaurant process (a Dirichlet process, in their words).
  The self-reinforcing property can be expressed informally as: more popular tables get more new customers, keeping them popular. A higher α means that customers are more likely to sit at a new table, so the distribution over tables will be more spread out in expectation. The distribution stabilizes as the number of iterations goes to infinity as an individual new customers has a diminishing effect the relative size of the weights.
  Algorithm 1 Expectation Chinese Restaurant Process
  1 assert type(alpha) is float and alpha > 0 2 assert type(n_iters) is int and n_iters >= 0 3 assert type(beta) is int and beta > 0 4 5 def sample_categorical_alpha(weights): 6 w_alpha = weights.copy() 7 k = num_nonzero(weights) 8 w_alpha[k + 1] = alpha 9 return sample_categorical(w_alpha / sum(w_alpha))
  10 11 weights = array([1, 0, 0, ...]) 12 for _ in range(n_iters): 13 addend = array([0, ...]) 14 for _ in range(beta): 15 i = sample_categorical_alpha(weights) 16 addend[i] += 1 / beta 17 weights += addend 18 return weights / sum(weights)
sections.heading: 4.2 EXPECTATION CHINESE RESTAURANT PROCESS
sections.text: The key difference between how the Chinese restaurant process and PPO works is the relationship between sampling (i.e., simulating episodes) and updating the weights/parameters. In each iteration, the regular Chinese restaurant process draws a sample based on its weights and updates those weight immediately. In PPO, the agent will populate the experience buffer with a number of steps (on the order of 100 to 1000) in the environment before performing gradient descent with that buffer to update the parameters. As a result, the parameter update is performed based on a weighting across multiple bottleneck units based on how often they were used in the episodes recorded in the experience buffer.
  Thus, to the appropriately generalize the Chinese restaurant process, we introduce the expectation Chinese restaurant process. In this process, we add a hyperparameter β which is a positive integer describing how many samples we take from the distribution before updating the weights; the updates are normalized by β so the sum of all weights still only increases by 1 per iteration. The restaurant analogy breaks down here as we would have to say that in each iteration, β customers simultaneously and independently make a decision, get shrunk to 1β th their size, and then sit at their table of choice. The pseudocode for the expectation Chinese restaurant process is given in Algorithm 1.
sections.heading: 5 EXPERIMENTS
sections.text: Each run of experiment starts by training a sender and receiver for a fixed number of timesteps for a range of independent variable values. The trained models are then evaluated by initializing 3000 episodes at evenly distributed locations using Vogel’s method (Vogel, 1979). In most settings, the agents are able to achieve a 100% success rate during training and evaluation; we remove any models which do not from consideration. All model for our experiments use 26 = 64 bottleneck units which translates to a maximum entropy of 6 bits. Hyperparameters are given in Appendix B.
sections.heading: 5.1 BIASED SEMANTICS
sections.text: In our first experiment we demonstrate how shaped rewards which are trivially biased directly distort the semantics of the language, that is, the action associated with each bottleneck unit. We compare three environments, no shaped rewards, the standard shaped reward, and the trivally biased shaped reward. We visualize the semantics of the language with so-called “sea star plots” in Figure 3. Each arm of the sea star is the action taken by the receiver in response to a single bottleneck unit with opacity representing the frequency of use.
  In the setting with no shaped rewards, we see actions (i.e., the meanings of the messages) learned featuring 2 to 4 arms pointing in a variety of directions. Since the standard shaped reward takes both dimensions into account, we do not see any bias in the direction of the learned actions. With the trivially biased reward, though, we see the the learned languages exclusively favor actions near to the horizontal axis. In this setting, nothing explicitly prevents the agents from learning vertical actions, but the fact that horizontal dimensions receive the shaped reward makes those actions easier to learn.
sections.heading: 5.2 CHANGING THE DISTRIBUTION OF ENTROPY
sections.text: Naturally, a shaped reward which favors certain actions over others will bias the semantics of the language. Thus, our second experiment investigates more closely the effect that shaped rewards without this explicit bias can have. Specifically, we investigate the distribution of language entropies in the two environments. By entropy we are specifically referring to the Shannon entropy (measured in bits) of the bottleneck units as used in the trained agents’ language (as averaged over 3000 quasirandomly initialized episodes).
  Entropy is an important aspect of language as it represents the upper bound on the information that the language can convey. Languages with higher entropy can yield more precise meaning in their utterances, yet this comes at the cost of being more difficult to learn or acquire as they need a greater variety of training example to be learned.
  To investigate the distribution of language entropies, we look at a histogram showing the Shannon entropy of languages belonging to environments with and without shaped rewards. The distributions is computed from 2000 independent runs for each reward setting. This is shown in Figure 4a. The presence of shaped rewards shifts the distribution upwards, demonstrating that even shaped rewards which is free of a trivial bias can still bias the emergent language. A potential explanation of these results is discussed and illustrated in Section 5.4.
sections.heading: 5.3 MASKING ENVIRONMENTAL PARAMETERS
sections.text: In our final primary experiment, we demonstrate how shaped rewards can mitigate the influence of environmental parameters on the entropy of the learned language. This is an issue insofar as the presence of shaped rewards make it difficult to observe an emergent property of interest. Specifically, we look at how the standard shaped rewards hide the effect of world radius on entropy in our centerward environment.
  In Figures 4b and 4c, we plot the language entropies against different world radii. In both settings, we observe that entropy decreases as the world radius increases, but the setting with no shaped rewards shows a much more rapid decrease in entropy. We offer one possible explanation for this effect in Section 5.4. When the only reward signal agents have access to is the base reward, they can only accomplish the task by finding the goal randomly at first; as the size of the environment increases, the chance of finding the goal with random movements decreases and the agent pair often fails to learn a fully successful language at the highest world radii.
sections.heading: 5.4 EXPERIENCE BUFFER SIZE
sections.text: As an explanatory experiment, we demonstrate how changing the size of the PPO experience buffer has a significant impact of the entropy of the emergent language. We compare this with the effects we would expect to see according to the model presented in the next section, i.e., the expectation Chinese Restaurant process. In turn we use this to explain one mechanism by which shaped rewards can have the observed effects on entropy shown by the previous experiments.
  In Figure 5a we show the effect of a logarithmic sweep of β on entropy of the expectation Chinese restaurant process. We first observe that increasing β reduces the variance between distributions yielded by the process since, as β increases, the individual updates made in each iterations are also reduced in variance. In fact, in the limiting case as β → ∞, the process will always yield the same distribution as the update will just be the expectation of sampling from the categorical distribution described by the (normalized) weights (plus α). The second effect is that increasing β will decrease the concentration (i.e., increase the entropy), on average, of the distribution yielded from the process. The intuition behind this is that the since each update is less concentrated, the distribution as a whole will be less concentrated as the probability mass will be spread out.
  These results can be used to explain, in part, both the effect of shaped rewards and world radius on entropy. First, though, we must establish an analogous correspondence between the expectation Chinese restaurant process and the PPO-learning process. An iteration of the process described in Algorithm 1 consists of sampling from the modified categorical distribution (Line 15) and incrementing the weights (Line 17). In PPO, the sampling corresponds to populating the experience buffer with steps associated with a reward, and the increment operation is analogous to PPO performing gradient descent on the agents using the buffer. Thus, β is analogously increased for PPO when the number of successful episodes per iteration which is dependent both on the size of the experience buffer as well as the environmental factors affecting frequency of success.
  In Figures 5b and 5c, we directly vary the size of the experience buffer in our environments with and without shaped rewards. Both environments replicate the correlation between β/buffer size and entropy, though the decrease in variance is less distinct as buffer size increases.
  Having established this correlation, we can offer a potential explanation to the experiments involving world radius as well as the distribution of entropies between the environments with and without shaped rewards. Shaped rewards effectively increase β since it assigns a reward signal to every step whereas the base reward-only environment requires successful episode. This effect is exacerbated when the world radius is increased, the base reward-only environment yields rewards less frequently in the beginning because randomly finding the goal is less likely. This effectively decreases β which corresponds to a lower entropy and higher variance.
sections.heading: 6 CONCLUSION
sections.text: We have, then, demonstrated the pitfalls that shaped rewards present for emergent language research: directly biasing the learned semantics of the language, changing the distribution of an emergent property of the language (i.e., entropy), and masking the emergent effects of other environmental variables. These experiments were performed with with a novel navigation-based emergent language environment. This environment allows allows for shaped rewards through multi-step episodes and avoids a trivially optimal language by employing a continuous state and action space. In addition to this, we introduced the expectation Chinese restaurant process both to explain our own experimental results and to provide a foundation for future models of emergent language.
  The limitations of this work can be illustrated through the future directions that could be taken. First, studying a variety of environments would further characterize what biases shaped rewards introduce into emergent languages. Second, increasing the complexity of environments would greatly expand the range of both emergent properties and types of shaped rewards which could be studied. For example, a rich environment like chess presents many emergent strategies such as the valuation of positions, balance between offense and defense, and favoring certain pieces; shaped rewards could then take the form of rewarding stochastic strategies, evaluating positions with traditional chess engines, or assigning pieces explicit point values. Furthermore, while we only use the expectation Chinese restaurant process as an explanatory model, further work could design experiments to demonstrate its predictive power.
  The studies presented in this paper are both exploratory and anticipatory in nature since emergent language research has yet to tackle environments difficult enough to require shaped rewards. Nevertheless, the field will follow reinforcement learning in tackling progressively more difficult tasks which will present further opportunities for or even require shaped rewards. When this occurs, accounting for the biases which are inherent to shaped rewards is imperative to preserving the integrity of emergent language experiments. This work, then, prepares researchers in the field for these future challenges.
sections.heading: REPRODUCIBILITY STATEMENT
sections.text: The code used in association with this paper is located at https://example.com/ repo-name (in a ZIP file for review). Reproduction instructions are located in the README.md file. Experiments were performed on a 20-thread Intel Core i9-9900X server on which they take less than 24 hours to run. No random seeds are recorded or provided as the training process is stable and similar results should be observed for any random seed.
sections.heading: A ENVIRONMENTS OF RELATED WORK
sections.text: Havrylov & Titov (2017) Standard signalling game where the observations are natural images from Microsoft COCO. There is no trivially optimal language since the information being communicated is simply which image is being shown. There are natural image classes (e.g., cat vs. dog), but they are not necessarily the features of the images which the agents need to communicate. The standard signalling game is single-step.
  Kottur et al. (2017) “Task and talk” dialog game where one agent must ask questions of the other agent to learn the attributes of an object. Specifically, the questioner has a set of attributes it must learn (the objective) about the object that only the answerer can see. The desired language has the questioner using a unique message to specify a property and the answerer responding with a unique message to specify the value for that property such that messages are always in one-to-one correspondence what they are communicating. Such a language is trivially optimal. Multi-round dialog environments are inherently multi-step.
  Mordatch & Abbeel (2018) Collaborative navigation game where multiple agents tell each other whither to move in a 2D environment. The environment consists of agents and landmarks distinguisted by color. Each agent is given a “goal”; for example, the blue agent might receive “red agent go to green landmark” (represented as a categorical feature vector) which the blue agent then has to communicate to the red agent. The agents have a vocabulary large enough to assign a unique “word” to each concept being expressed in one-to-one correspondence, yielding a trivially optimal language. Each episodes consists of multiple timesteps which can each have their own utterance.
  Lazaridou et al. (2018) An image-based signalling game using rendered images from MuJoCo. The information being communicated is the shape and color of an object where set of shapes and colors are both small. Since the number of words available is at least as big as the cardinalities of the sets and the sender is able to use multiple words per utterance, it is possible to construct a trivially optimal langauge. The standard signalling game is single-step.
  Kharitonov et al. (2020) A binary vector-based signalling game with shared context. The goal of the signalling game is to communicate the bits of a binary vector which are not shared by the sender and receiver. The messages consist of a single symbol where the number of unique symbols is greater than the combination of bits to communicated, thus there is a trivially optimal langauge where one unique symbol is assigned to each possible combination of unshared bits. The standard signalling game is single-step.
  Chaabouni et al. (2021) A signalling game with 330 different colors represented as a real vector in CIELAB color space. The game is set up so that colors which are nearby never appear in the same distractor set which encourages solutions which can cover some arbitrary region of colors. Due to this “fuzzy” concept of solution (i.e., not using 330 distinct words for each color), we consider this environment not to have a trivially optimal solution. The standard signalling game is single-step.
sections.heading: B HYPERPARAMETERS
sections.heading: B.1 DEFAULT CONFIGURATION
sections.text: Environment
  • Type: centerward • World radius: 9 • Goal radius: 1 • Max steps per episode: 3× world radius
sections.heading: Agent Architecture
sections.text: • Bottleneck size: 64 • Architecture; sender is 1-3 and receiver is 5; bottleneck size is N
  1. Linear w/ bias: 2 in, 32 out 2. Tanh activation 3. Linear w/ bias: 32 in, N out 4. Gumbel-Softmax: N in, N out 5. Linear w/ bias: N in, 2 out (action) and 1 out (value)
  • Bottleneck (Gumbel-Softmax) temperature: 1.5 • Weight initialization: U ( − √ 1 n , √ 1 n ) , where n is the input size of the layer (PyTorch 1.10
  default)
sections.heading: Optimization
sections.text: • Reinforcement learning algorithm: proximal policy optimization – Default hyperparameters used unless otherwise noted: https:// stable-baselines3.readthedocs.io/en/v1.0/modules/ppo. html
  • Training steps: 1× 105
  • Evaluation episodes: 3× 103
  • Learning rate: 3× 10−3
  • Experience buffer size: 1024
  • Batch size: 256 • Temporal discount factor (γ): 0.9
sections.heading: B.2 EXPERIMENT-SPECIFIC CONFIGURATIONS
sections.text: Note that we define a logarithmic sweep from x to y (inclusive) with n steps to be defined by Equation 3. {
  x · (y x ) i n−1 ∣∣∣∣ i ∈ {0, 1, . . . , n− 1}} (3) Biased Semantics
  • Type: edgeward • World radius: 8 • Goal Radius: 8 • Experience buffer size: 256 • Batch size: 256
sections.heading: Changing the Distribution of Entropy
sections.text: • Number of independent runs per configuration: 2000 • Experience buffer size: 256 • Batch size: 256
sections.heading: Masking Environmental Parameters
sections.text: • World radius: logarithmic sweep from 2 to 40 with 300 steps
sections.heading: Experience Buffer Size
sections.text: • Experience Buffer Size: logarithmic sweep from 32 to 4096 with 200 steps (floor function applied as experience buffer size is integer-valued)
  • Training steps: 2× 105
sections.heading: Expectation Chinese Restaurant Process
sections.text: This is not an emergent language experiment and just consists of running the mathematical model.
  • α: 5 • β: logarithmic sweep from 1 to 1000 with 10 000 steps (floor function applied, though
  plotted without floor function) • Number of iterations per run: 1000
title: SHAPED REWARDS BIAS EMERGENT LANGUAGE