summaries: The paper proposes Rational Inattention Reinforcement Learning (RIRL), a multi-agent reinforcement learning framework that incorporates the behavioral model of rational inattention. Building on the classical formalization of rational inattention, the objective penalizes for the mutual information between the action selected and the observable state of the world. They also extend their framework to support multiple channels of information with heterogeneous cognitive costs.   They empirically evaluate RIRL in two principal-agent problems, building on classical models from the economics literature. The first simulation consists of a single agent, while the second simulation explores interactions between multiple agents following the RIRL-actor architecture (although only the principal faces inattention).  Standard Bayesian-type decision making models have that given some signal X, our decision makers choose an action A to maximize utility given the signal X. This gives a conditional distribution A|X. Rational inattention models say that the there is limted processing capacity in the agent and model this by requiring that A | X has a capacity constraint (i.e. which under some extra assumptions essentially becomes maximum amount of mutual information).   The authors put this into RL by, basically, writing a policy P(a|s) into an observation component (O(observation | state)) and an action conditional on the realized observation with an information penalty (so the higher capacity the channels, the more penalty).   The authors apply this RIRL model to some canonical game theory games. The paper proposes a RIRL, which incoporate rational inattention suboptimality into the optimal RL model. The authors simulate RIRL in a single agent and a multi-agent setting. They compare RIRL behavior with the rational model in those settings and discuss the emergent behavior of RIRL.  This paper studies human bounded rationality using multi-agent reinforcement learning. It particularly focuses on the rational inattention model and proposes the RIRL framework which allows multi-timestep dynamics and information channels with heterogeneous processing costs. The proposed RIRL framework is evaluated in two specific Principal-Agent problems. 
abstractText: Multi-agent reinforcement learning (MARL) is a powerful framework for studying emergent behavior in complex agent-based simulations. However, RL agents are often assumed to be rational and behave optimally, which does not fully reflect human behavior. Here, we study more human-like RL agents which incorporate an established model of human-irrationality, the Rational Inattention (RI) model. RI models the cost of cognitive information processing using mutual information. Our RIRL framework generalizes and is more flexible than prior work by allowing for multi-timestep dynamics and information channels with heterogeneous processing costs. We evaluate RIRL in Principal-Agent (specifically manager-employee relations) problem settings of varying complexity where RI models information asymmetry (e.g. it may be costly for the manager to observe certain information about the employees). We show that using RIRL yields a rich spectrum of new equilibrium behaviors that differ from those found under rational assumptions. For instance, some forms of a Principal’s inattention can increase Agent welfare due to increased compensation, while other forms of inattention can decrease Agent welfare by encouraging extra work effort. Additionally, new strategies emerge compared to those under rationality assumptions, e.g., Agents are incentivized to misrepresent their ability. These results suggest RIRL is a powerful tool towards building AI agents that can mimic real human behavior.
authors.name: RATIONALLY INATTEN
references.authors: Sanjeevan Ahilan Peter Dayan
references.title: Feudal multi-agent hierarchies for cooperative reinforcement learning
references.venue: Workshop on “Structure & Priors in Reinforcement Learning”
references.authors: Bowen Baker Ingmar Kanitscheider Todor Markov Yi Wu Glenn Powell Bob McGrew Igor Mordatch
references.title: Emergent tool use from multi-agent autocurricula
references.authors: Ruairidh M Battleday Joshua C Peterson Thomas L Griffiths
references.title: Modeling human categorization of natural images using deep feature representations
references.venue: arXiv preprint arXiv:1711.04855,
references.authors: Mohamed Ishmael Belghazi Aristide Baratin Sai Rajeswar Sherjil Ozair Yoshua Bengio Aaron Courville R Devon Hjelm
references.title: Mine: mutual information neural estimation
references.authors: Eric Bonabeau
references.title: Agent-based modeling: Methods and techniques for simulating human systems
references.venue: PNAS,
references.authors: David D Bourgin Joshua C Peterson Daniel Reichman Stuart J Russell Thomas L Griffiths
references.title: Cognitive model priors for predicting human decisions
references.venue: In International conference on machine learning,
references.authors: Aysun Bozanta Aslıhan Nasır
references.title: Usage of agent-based modeling and simulation in marketing
references.venue: Journal of Advanced Management Science Vol,
references.authors: Vı́ctor Campos Alexander Trott Caiming Xiong Richard Socher Xavier Giró-i Nieto Jordi Torres
references.title: Explore, discover and learn: Unsupervised discovery of state-covering skills
references.venue: In International Conference on Machine Learning,
references.authors: Mark Dean Nate Leigh Neligh
references.title: Experimental tests of rational inattention
references.authors: Mathias Erlei Heike Schenk-Mathes
references.title: Bounded rationality in principal-agent relationships
references.venue: German Economic Review,
references.authors: Benjamin Patrick Evans Mikhail Prokopenko
references.title: Bounded rationality for relaxing best response and mutual consistency: The quantal hierarchy model of decision-making
references.venue: arXiv preprint arXiv:2106.15844,
references.authors: Benjamin Eysenbach Abhishek Gupta Julian Ibarz Sergey Levine
references.title: Diversity is all you need: Learning skills without a reward function
references.authors: Xavier Gabaix
references.title: Behavioral Inattention
references.venue: In Handbook of Behavioral Economics
references.authors: Corrado W Gini
references.title: Variability and mutability, contribution to the study of statistical distributions and relations. studi cconomico-giuridici della r. universita de cagliari (1912). reviewed in: Light, rj, margolin, bh: An analysis of variance for categorical data
references.venue: J. American Statistical Association,
references.authors: Jordi Grau-Moya Felix Leibfried Peter Vrancx
references.title: Soft q-learning with mutual-information regularization
references.venue: In ICML,
references.authors: Karol Gregor Danilo Jimenez Rezende Daan Wierstra
references.title: Variational intrinsic control
references.venue: arXiv preprint arXiv:1611.07507,
references.authors: Sanford J Grossman Oliver D Hart
references.title: An analysis of the principal-agent problem
references.venue: In Foundations of insurance economics,
references.authors: Jason Siyanda Hartford
references.title: Deep learning for predicting human strategic behavior
references.venue: PhD thesis, University of British Columbia,
references.authors: Joseph G Haubrich
references.title: Risk aversion, performance pay, and the principal-agent problem
references.venue: Journal of Political Economy,
references.authors: Sepp Hochreiter Jürgen Schmidhuber
references.title: Long Short-Term Memory
references.venue: Neural Computation,
references.authors: William Hoiles Vikram Krishnamurthy Kunal Pattanayak
references.title: Rationally inattentive inverse reinforcement learning explains youtube commenting behavior
references.venue: J. Mach. Learn. Res.,
references.authors: Bengt Holmstrom Paul Milgrom
references.title: Aggregation and Linearity in the Provision of Intertemporal Incentives
references.authors: Gege Jiang Mogens Fosgerau Hong K Lo
references.title: Route choice, travel time variability, and rational inattention
references.venue: Transportation Research Procedia,
references.authors: Kris N Kirby Richard J Herrnstein
references.title: Preference reversals due to myopic discounting of delayed reward
references.venue: Psychological science,
references.authors: Yoav Kolumbus Gali Noti
references.title: Neural networks for predicting human interactions in repeated games
references.venue: arXiv preprint arXiv:1911.03233,
references.authors: Dmytro Korzhyk Vincent Conitzer Ronald Parr
references.title: Complexity of computing optimal stackelberg strategies in security resource allocation games
references.venue: In Twenty-fourth aaai conference on artificial intelligence,
references.authors: Jonas Kubilius Martin Schrimpf Kohitij Kar Ha Hong Najib J Majaj Rishi Rajalingham Elias B Issa Pouya Bashivan Jonathan Prescott-Roy Kailyn Schmidt
references.title: Brain-like object recognition with high-performing shallow recurrent anns
references.venue: arXiv preprint arXiv:1909.06161,
references.authors: Maciej Łatek RL Axtell Bogumil Kaminski
references.title: Bounded rationality via recursion
references.venue: In Proceedings of Eighth International Conference on Autonomous Agents and Multi-Agent Systems (AAMAS
references.authors: Felix Leibfried Daniel A. Braun
references.title: Bounded rational decision-making in feedforward neural networks
references.venue: In UAI,
references.authors: Felix Leibfried Jordi Grau-Moya
references.title: Mutual-information regularization in markov decision processes and actor-critic learning
references.venue: In CoRL,
references.authors: Felix Leibfried Jordi Grau-Moya Haitham Bou-Ammar
references.title: An Information-Theoretic Optimality Principle for Deep Reinforcement Learning
references.venue: NeurIPS Workshop on Deep Reinforcement Learning,
references.authors: Joel Z. Leibo Vinicius Zambaldi Marc Lanctot Janusz Marecki Thore Graepel
references.title: Multi-agent reinforcement learning in sequential social dilemmas
references.authors: Wei Ji Ma Benjamin Peters
references.title: A neural network walks into a lab: towards using deep nets as models for human
references.authors: Bartosz Maćkowiak Mirko Wiederholt
references.title: Busines Cycle Dynamics Under Rational Inattention
references.venue: European Central Bank Working Paper Series,
references.authors: Bartosz Mackowiak Filip Matejka Mirko Wiederholt
references.title: Rational inattention: A review
references.authors: Gary J Miller
references.title: The political evolution of principal-agent models
references.venue: Annu. Rev. Polit. Sci.,
references.authors: James A Mirrlees
references.title: The optimal structure of incentives and authority within an organization
references.venue: The Bell Journal of Economics,
references.authors: Shakir Mohamed Danilo Jimenez Rezende
references.title: Variational information maximisation for intrinsically motivated reinforcement learning
references.venue: NeurIPS,
references.authors: Pedro A. Ortega Daniel A. Braun Justin Dyer Kee-Eung Kim Naftali Tishby
references.title: InformationTheoretic Bounded Rationality
references.venue: arXiv preprint arXiv:1512.06789,
references.authors: Zhen Peng Tim Genewein Felix Leibfried Daniel A Braun
references.title: An information-theoretic on-line update principle for perception-action coupling
references.venue: IEEE/RSJ International Conference on intelligent robots and systems (IROS),
references.authors: John W Pratt
references.title: Risk aversion in the small and in the large
references.venue: In Uncertainty in economics,
references.authors: Ardeshir Raihanian Mashhadi Sara Behdad
references.title: Environmental impact assessment of the heterogeneity in consumers’ usage behavior: An agent-based modeling approach
references.venue: Journal of Industrial Ecology,
references.authors: Yuliy Sannikov
references.title: A continuous-time version of the principal-agent problem
references.venue: The Review of Economic Studies,
references.authors: Zhenyu Shi Runsheng Yu Xinrun Wang Rundong Wang Youzhi Zhang Hanjiang Lai Bo An
references.title: Learning expensive coordination: An event-based deep rl approach
references.venue: In ICLR,
references.authors: Yoav Shoham Kevin Leyton-Brown
references.title: Multiagent systems: Algorithmic, game-theoretic, and logical foundations
references.authors: Tianmin Shu Yuandong Tian
references.title: M3rl: Mind-aware multi-agent management reinforcement learning
references.authors: Herbert A Simon
references.title: Models of man; social and rational
references.authors: Christopher A Sims
references.title: Implications of rational inattention
references.venue: Journal of monetary Economics,
references.authors: Michael Spence
references.title: Job Market Signaling
references.venue: The Quarterly Journal of Economics,
references.authors: Klaus Spremann
references.title: Agent and Principal
references.venue: Agency Theory, Information, and Incentives,
references.authors: Susanne Still Doina Precup
references.title: An information-theoretic approach to curiosity-driven reinforcement learning
references.venue: Theory in Biosciences,
references.authors: Richard S. Sutton Andrew G. Barto
references.title: Reinforcement learning: An introduction
references.venue: MIT press,
references.authors: Alexander Trott Sunil Srinivasa Douwe van der Wal Sebastien Haneuse Stephan Zheng
references.title: Building a Foundation for Data-Driven, Interpretable, and Robust Policy Design using the AI Economist
references.venue: arXiv preprint arXiv:2108.02904,
references.authors: Amos Tversky Daniel Kahneman
references.title: Advances in Prospect Theory: Cumulative Representation of Uncertainty
references.venue: Journal of Risk and Uncertainty,
references.authors: Ying Wen Yaodong Yang Rui Luo Jun Wang
references.title: Modelling bounded rationality in multi-agent interactions by generalized recursive reasoning
references.authors: Ronald J Williams
references.title: Simple statistical gradient-following algorithms for connectionist reinforcement learning
references.venue: Machine learning,
references.authors: Yaodong Yang Lantao Yu Yiwei Bai Jun Wang Weinan Zhang Ying Wen Yong Yu
references.title: A Study of AI Population Dynamics with Million-agent Reinforcement Learning
references.authors: Stephan Zheng Alexander Trott Sunil Srinivasa Nikhil Naik Melvin Gruesbeck David C Parkes Richard Socher
references.title: The ai economist: Improving equality and productivity with ai-driven tax policies
references.authors: Stephan Zheng Alexander Trott Sunil Srinivasa David C. Parkes Richard Socher
references.title: The AI Economist: Optimal Economic Policy Design via Two-level Deep Reinforcement Learning
references.venue: arXiv preprint arXiv:2108.02755,
sections.heading: 1 INTRODUCTION
sections.text: Multi-agent reinforcement learning (MARL) has shown great utility in complex agent-based model (ABM) simulations, e.g., in economics, games, and other fields (Zheng et al., 2020; Baker et al., 2020; Leibo et al., 2017; Yang et al., 2018). ABM is a field of research which has been found important and useful for quantifying the potential impacts of various policies in diverse real world applications (eg. market (Bozanta & Nasır, 2014) or environmental impact (Raihanian Mashhadi & Behdad, 2018) simulations) by examining real-world systems through simulation. In many of such simulations, the behavioral rules of agents may be too difficult for designers to specify. With MARL, designers instead specify the agents’ objective functions and the reinforcement learning (RL) agents autonomously learn behaviors to optimize their objectives. However, this approach may be problematic when simulating systems of human agents, because in contrast to established models of human decision-making, it assumes that agents behave rationally and execute the objective-maximizing behavior. Prior literature demonstrates that using established models of human-irrationality during decision making yields results and implications that are significantly different from those obtained using rationality assumptions (Sims, 2003; Maćkowiak & Wiederholt, 2011; Jiang et al., 2019). Therefore, it is important to account for this irrationality when simulating systems involving human(-like) agents.
  To this end, we introduce Rational Inattention Reinforcement Learning (RIRL), a MARL framework with agents that can be rationally inattentive. Rational Inattention (RI) is a model of bounded rationality (Sims, 2003) that is well established in behavioral economics (Mackowiak et al., 2021). It attributes human irrationality to the costliness of mental effort (e.g. attention) required to identify the optimal action. Mathematically, the RI framework measures these costs as the mutual information (MI) between the variables considered by the decision process and the decisions ultimately made.
  This captures the intuition that a policy has a higher cognitive cost if its execution requires more information about the state of the world and thus more attention. MI-based rewards have been used in RL, (e.g. Leibfried & Braun (2016); Leibfried et al. (2018); Grau-Moya et al. (2018); Leibfried & Grau-Moya (2020) use RI-style MI costs to regularize learning) but not to model sub-optimal behavior. When used to model sub-optimal behavior, RIRL can rationalize seemingly sub-optimal behavior by including this cognitive cost in the reward function, i.e., by adding the MI cost(s). That is, the “rational” behaviors of an RIRL-actor can mimic human-like bounded rationality.
  Contributions. The main contributions of this work are: (1) we propose RIRL, a framework for modeling bounded-rationality which can analyze complex scenarios not previously possible; and (2) we validate RIRL’s modeling capabilities in classical economic settings intractable using the analytical methods of prior work. To our knowledge, we are the first to study the effects of Rational Inattention in MARL-based simulations.
  RIRL extends the single-timestep framework proposed by Peng et al. (2017), which decomposes decision-making into stochastic perception followed by stochastic action, each subject to its own MI cost. RIRL provides a novel boundedly-rational policy class that generalizes this to multiple information channels with heterogeneous costs, hidden-state policy models, and sequential environments. To achieve this increased modeling power, our architecture uses methods significantly different from prior work, such as using deep mutual information estimation modules to allow for multiple channels and using an LSTM to account for multiple timesteps. This allows RIRL to analyze settings with rich cognitive cost structures, e.g., when information about state variables have different observation difficulty. For example, in hiring, a candidate’s past job performance may be more relevant but harder to evaluate than her employment history.
  We evaluate RIRL in two Principal-Agent (PA) problems, specifically manager-employee relations, where the Principal is boundedly rational. We use RIRL to model asymmetric information (i.e. hard to obtain information) between the Principal and the Agent they are incentivizing. Real-world PA experiments have shown that bounded rationality is key to explaining marked deviations between equilibria reached by human participants and theoretical predictions (Erlei & Schenk-Mathes, 2017). Prior work considered bounded rationality assumptions but found it to be analytically difficult (Mirrlees, 1976) and further research in this direction is sparse
  We show that RIRL allows us to analyze generalized PA problems that are analytically intractable, such as a sequential PA problem with multiple Agents and heterogeneous information channels. Across all settings, we observe that equilibrium implications depends strongly on the cost(s) of attention and differs from that under rational assumptions sometimes in ways that are hard to intuitively predict. We observe that, depending on the channel, increasing Principal inattention can either increase Agent welfare due to increased compensation or decrease Agent welfare due to encouraging additional work. Compared to under rational assumptions, we additionally observe the emergence of different strategies, such as Agents choosing to misrepresent their ability, classically referred to as signaling (Spence, 1973). Our results show that RIRL can be a powerful tool to model boundedly rational behavior and analyze its emergent consequences in multi-agent systems.
sections.heading: 2 RELATED WORKS
sections.text: Multi-Agent Reinforcement Learning for Agent-Based Simulation. Agent-Based Models (ABM) are a popular tool use to study real-world systems (e.g., organizations or market systems) and discover emergent behavior through simulation with fixed or manually-specified agent behaviors (Bonabeau, 2002). Instead, Multi-Agent Reinforcement Learning (MARL)-based simulations use RL agents which autonomously learn utility-maximizing behavior, so designers do not need to specify behavioral rules. MARL has been used to study tax policy (Zheng et al., 2020; 2021; Trott et al., 2021), games (Baker et al., 2020), and social dilemmas (Leibo et al., 2017) among others (Yang et al., 2018). However, RL agents are mostly assumed rational which contradicts how humans make decisions. Some recent work considers bounded rationality in MARL by accounting for cognitive limitations when reasoning about the behaviors of other agents (Evans & Prokopenko, 2021; Wen et al., 2019; Łatek et al., 2009). We model a complementary source of bounded rationality: the cognitive costs of attending to all available information when forming a decision.
  Models of Irrationality. Behavioral economics has shown extensively that human decision-making is not fully rational, but instead features many cognitive biases (Caverni et al., 1990). Bounded rationality (Simon, 1957) attributes these human irrationalities to resource limitations, e.g., bounded cognitive capabilities or costliness of using (more) time to make decisions. Rational Inattention (RI) is a well-established model of bounded rationality which models the cognitive cost of decisions as the mutual information between variables relevant to the decision and the decision itself (Sims, 2003; Mackowiak et al., 2021). RI has been tested in real world experiments (Dean & Neligh, 2017) and used to model human behavior in a wide variety of domains (Hoiles et al., 2020; Mackowiak et al., 2021). Closest to our work, Jiang et al. (2019) study a multi-agent system for traffic route choice under RI but where the agents do not react to each other’s actions.
  Another line of work uses human behavioral data with deep learning to model human cognition (Kubilius et al., 2019; Battleday et al., 2017; Ma & Peters, 2020) or predict human behavior (Bourgin et al., 2019; Kolumbus & Noti, 2019; Hartford, 2016). In contrast, our ABM approach examines the implications of RI in complex settings where experimental data are presently unavailable.
  Mutual Information in Reinforcement Learning. MI has been extensively studied in the intrinsically motivated RL literature for curiosity-driven exploration and unsupervised skill and option discovery (Still & Precup, 2012; Campos et al., 2020; Mohamed & Rezende, 2015; Gregor et al., 2016; Eysenbach et al., 2019), often with differing techniques used to measure MI. Recently MI-based rewards have been used to regularize exploration (Grau-Moya et al., 2018; Leibfried & Grau-Moya, 2020). Comparatively, very little work has considered MI-based rewards for modeling boundedly rational behavior (Ortega et al., 2015; Peng et al., 2017), with no prior work, to our knowledge, considering the domain of multi-agent simulations.
  Principal-Agent Problems. In addition to our guiding example, Principal-Agent problems have been used to describe various settings, including politics and insurance (Miller, 2005; Grossman & Hart, 1992). Prior economics literature mainly use analytical methods and narrow modeling assumptions, e.g., that all stochasticity follows Brownian motion (Sannikov, 2008) or certain separability conditions are satisfied (Grossman & Hart, 1992). Generally, Principal-Agent settings fall under the category of Stackelberg games where it has been shown computing optimal strategies can be NP-hard (Korzhyk et al., 2010). Instead, MARL can study complex setups that are analytically intractable. Shu & Tian (2019); Shi et al. (2019); Ahilan & Dayan (2019) studied coordinating cooperation in a Principal-Agent model, with an RL Principal learning to incentivize Agents to achieve an overall goal. However, they do not consider bounded rationality.
sections.heading: 3 PRELIMINARIES
sections.text: We consider multi-agent simulations in the form of partially-observable Markov Games (MGs), formally defined by (S,A, r, T , γ, O, I) (Sutton & Barto, 2018). Here S is the state space of the game, A is the combined action spaces of the actors, and I are actor indices. We use oi = O(s, i) to denote the portion of the game state s that actor i can observe. In addition, oi may include a (possibly learnable) encoding of the observation history. Each game episode has a horizon of T ≥ 1 timestep(s). Each timestep t, actor i selects action ai,t, sampling from its policy πi (ai,t|oi,t). Given the sampled actions, the transition function T determines how the state evolves. Each actor’s objective is encoded in its reward function ri(s,a), (boldface denotes concatenation across actors). When modeling economic behavior, the reward is the (marginal) utility Ui(st,at), an abstract measure of happiness. Each actor optimizes its policy to maximize its γ-discounted sum of future rewards.
  While RL can be used to discover (approximately) utility-maximizing policies, such rational behavior fails to account for characteristic irrational human behavior. Behavioral economic models often model irrationality as consequences of inattention (Gabaix, 2017). Rational Inattention formalizes this intuition using a modified objective that includes a cost to the mutual information I(ai; oi) between the (observable) state of the world oi and the actions ai ∼ πi(·|oi). This definition captures the intuition that if the agent puts in more effort to pay attention to oi, its action ai likely becomes more correlated with the observation oi, and thus high MI.
  π†i = argmax πi (Eπ [Ui(s,a)]− λI(ai; oi)) . (1)
  Note that this is equivalent to learning the optimal policy for an adjusted reward function: r†i (st,at) = Ui(st,at) − λĨ(ai,t; oi,t), where Ĩ(ai,t; oi,t) = log p(ai,t, oi,t) − log p(ai,t)p(oi,t) is a Monte Carlo estimate of I(ai; oi), and λ is the utility cost per bit of information. Here p(ai, oi), p(ai), and p(oi) are the joint and marginal distributions over ai and oi induced by the environment and policies π. Below, we describe our method for estimating these quantities during training.
sections.heading: 4 MODELING AND TRAINING BOUNDEDLY RATIONAL ACTORS WITH RIRL
sections.text: MI Estimation. RIRL uses a general-purpose module for estimating Ĩπi(ai; oi), i.e., the singlesample MC estimate of the mutual information between oi and ai ∼ πi(ai|oi). Given the pair (ai, oi), we estimate Ĩπi(ai; oi) from the ratio between log p(ai, oi) (the log-odds under the joint distribution) and log p(ai)p(oi) (the log-odds under the factorized distributions). This ratio can be estimated using a discriminator dπi(ai, oi) that learns to classify which of the two distributions the sample (ai, oi) came from. Samples from p(ai, oi) are generated during on-policy rollouts, while samples from p(ai)p(oi) can be generated by shuffling a batch of samples from the joint distribution. As such, on-policy rollout data can be used both to optimize π and to train dπi , and compute r†i,t = ri,t − λĨπi(ai,t; oi,t). While we favor the above approach for its simplicity, other techniques for estimating MI have also been proposed, e.g. Belghazi et al. (2018). Our framework is not restricted to the choice of MI estimator used here, but a comparison of these techniques is outside the scope of the current work.
  Action-Perception Decoupling with Multiple Information Channels. Penalizing I(ai; oi) models the intuition that it is costly to obtain or use information about the world, e.g., to reduce uncertainty about oi,t or to identify the utility-maximizing action given oi,t. To support richer modeling, we also extend our framework with multiple channels of information with heterogeneous cognitive costs. For example, when buying a used car it is easier to see car prices than their actual condition.
  To that end, we extend the “action-perception decoupling” strategy of Peng et al. (2017), which models π(a|s) as a stochastic perception module q(y|s) followed by an action module p(a|y), jointly trained to optimize an RI-style reward r(s, a)−λqIq(y; s)−λpIp(a; y). We extend this using a policy class that can flexibly model M information channels (i.e., partitions of o) with different processing costs λm. We also use recurrent policies which may allocate processing costs strategically over time.
  More formally, we assume that an observation ot can be decomposed as a set ofM ≥ 1 observations ot = {o1t , . . . , oMt }, with omt being the observation from information channel m. In addition, we assume that each channel has an associated information cost: λ = {λ1, . . . , λM}. For each channel, we learn an encoder fm(ymt |omt , ψt), which takes omt and recurrent state ψt (see below) as inputs and outputs the parameters (means and standard deviations) of a stochastic encoding ymt . In practice, we implement each fm as a residual-style encoder, with samples given by:
  µmt , σ m t = f m(omt , ψt), y m t = o m t + µ m t + σ m t · ϵmt , (2)
  where ϵmt is a random sample from a spherical Gaussian with dimensionality equal to that of ym and om. For each fm, we also train a discriminator dfm(ymt , [o m t , ψt]) used to estimate Ĩfm(y m t ; [o m t , ψt]). The full encoding yt = [ y1t , . . . , y M t ] of ot concatenates all M encoder samples. Before sampling an action, we use an LSTM (Hochreiter & Schmidhuber, 1997) to maintain a history of (encoded) information: ψt+1 = LSTM(yt, ψt). The encoding yt and updated LSTM state ψt+1 are used as inputs to the stochastic action module ω(at|yt, ψt+1) which outputs a distribution over actions.
  Training. We train this architecture with policy gradients (Williams, 1992): ∆π ∝ E ( ∇ log π(y1t , . . . , yMt , at|st, ψt, ψt+1)r † t ) , (3)
  log π(y1t , . . . , y M t , at|st, ψt, ψt+1) = logω(at|yt, ψt+1) + M∑ m=1 log fm(ymt |omt , ψt), (4)
  r†t = U(st,at)− λω Ĩω(at; [yt, ψt+1])− M∑
  m=1
  λmĨfm(y m t ; [o m t , ψt]). (5)
sections.heading: 5 VALIDATING RIRL IN PRINCIPAL-AGENT PROBLEMS
sections.text: We demonstrate RIRL’s modeling capabilities in two Principal-Agent (PA) problems of varying complexity. In both settings, a boundedly rational manager (the Principal) decides how to compensate its workers (the Agents). In general, labor is costly to the Agents but beneficial to the Principal, while the Agents benefit from income that is costly for the Principal to provide. Therefore, the Principal aims to find a wage schedule W that maximizes its profits. PA problems often consider how information asymmetry between the Principal and Agent(s) influences equilibrium schedules. RIRL can analyze a meaningful extension in which we model how the cost of information influences equilibrium schedules, i.e., where the Principal can spend effort to reduce information asymmetry.
  We first study a bandit-style PA setting with a Principal that optimizes a wage schedule for each level of Agent output, subject to a cognitive cost for observing output. Because some aspects of the solution are tractable with prior analytical methods, this simple setting provides a useful validation of RIRL in a multi-Agent game. However note, even in this simple setting, aspects of the solution (such as the constants) are not tractable analytically. Secondly, we apply RIRL to analyze a complex sequential PA problem with multiple Agents and heterogeneous information channels that is outside the scope of analytical methods of prior work.
  Our experimental results were able to uncover several non-trivial insights when bounded rationality is assumed that are hard to intuitively predict, demonstrating the importance of modeling bounded rationality in such simulations. Insights include (1) Agent utility can increase with principal inattention and a more rational Agent benefits more from Principal inattention due to the Principal setting higher pay schedules. To our knowledge, no prior work has identified the consequences of bounded rationality in both the Principal and Agent. (2) In the case of where there are multiple Agents, a boundedly rational Principal can induce Prisoner’s Dilemma like incentives for the Agents, reducing the welfare of all agents. We discuss these insights along with others in detail below.
sections.heading: 5.1 PRINCIPAL-AGENT WITH A SINGLE AGENT, SINGLE TIMESTEP (BANDIT SETTING)
sections.text: We first explore RIRL in a PA setting with a single Agent, following classic work (Mirrlees, 1976; Holmstrom & Milgrom, 1987; Spremann, 1987; Haubrich, 1994). The Agent’s labor output z ∼ h(e) is a stochastic function of its effort action e. The Agent chooses how much to work based on its payment schedule W , which is controlled by the Principal and yields payment w as a stochastic function of output z, i.e., w ∼ W(z) = N (µz, σz). Here, the stochasticity in the pay schedule aims to model the Principal’s uncertainty over z, e.g., due to limited attention. By assumption, the Agent accurately anticipates the shape of the payment schedule W when making its decisions, e.g., based on prior experience or reputation. As such, the RIRL-Principal sets the payment schedule:
  W∗ = argmax W [E [Up(w, z)]− λIW(w; z)]w∼W(z),z∼h(e),e∼πβa (e|W) . (6)
  Here, λIW(w; z) is the attention cost for W . For the Agent, we use a soft-Q policy πβa (e|W) = eβUa(e|W)/Z with normalization Z, where Ua(e|W) denotes the expected utility of effort e given schedule W . Note that πβa can be calculated directly. Importantly, the soft-Q policy models a form of bounded rationality, i.e., the log-odds of effort e is proportional to its expected utility, with higher β making “better” actions e more likely. Hence, low β corresponds to less-than-rational behavior.
  The utility of the Principal (Up) and Agent (Ua) follow standard economic utility functions:
  Ua(w, e) = CRRA(w; ρ)︸ ︷︷ ︸ Income Utility − e︸︷︷︸ Work Disutility and Up(w, z)︸ ︷︷ ︸ Profit = z︸︷︷︸ Revenue − w︸︷︷︸ Amount Paid
  (7)
  Here CRRA is the concave Constant Relative Risk Adverse function (Pratt, 1978); the risk aversion parameter ρ sets its concavity (we use ρ = 2). More environment details are in the Appendix. We use N -sample policy gradients on estimates of Equation 6 to learn the parameters µz and σz of W∗
  r†W = 1
  N N∑ i=1 [ Up(wi, zi)− λĨW(wi; zi) ] , wi ∼ W(zi), zi ∼ h(ei), ei ∼ πβa (e|Wi). (8)
  Results. Figure 1 shows the effects of the Principal’s inattention on simulated outcomes. We also compare RIRL outcomes against those found including entropy-based rewards. MI-based and
  entropy-based regularization are closely related (Grau-Moya et al., 2018; Leibfried & Grau-Moya, 2020), with subtle but important differences1. Nevertheless, they yield markedly different outcomes.
  To highlight an important difference, consider the constant 0-pay schedule. This provides no incentive for the Agent to work and costs the Principal nothing, and it therefore provides a reasonable lower bound on Up. This lower bound should hold under bounded rationality, since no attention is required when W treats all outputs identically. Indeed, under RIRL, Up approaches this lower bound as increasing λ leads the Principal to trade more profitable W’s for ones with smaller demands on attention (Figs. 1a, b, 2c). In contrast, under entropy regularization, increasing λ quickly yields W’s that violate the Up lower bound (Fig. 1a, c).
  Figure 2 shows how Agent and Principal utility and I(w; z) change as a function of λ, across a range of Agent β’s. Note that higher β increases the odds that the Agent selects the optimal action. This reveals an interesting multi-Agent interaction. As before, we observe that the Principal’s utility Up (without the attention cost) decreases with increasing λ. Furthermore, Up is generally lower when β is lower (Fig. 2b), as the Principal must pay more to influence πβa towards different actions.
  Interestingly, we observe that the Agent’s utility Ua can actually increase when the Principal is inattentive, and that higher β tends to increase the beneficial range of inattention (Fig. 2a). Lastly, β determines how the Principal responds to increasing attention costs. Agents that behave optimally more often (higher β) incentivize the Principal more to pay attention (Fig. 2c).
  Comparison With Prior Literature. For this setting, Mirrlees (1976) found the theoretically optimal pay schedule should be of the form W(z) = max(Az +B,C) 1 ρ . While they assume a slightly
  1Under some conditions, they are identical. However, the key difference is that entropy regularization penalizes the policy for deviating from a fixed, uniform prior, whereas RI-style MI regularization penalizes deviations from an optimal prior.
  different model of costly attention2, the emergent pay schedules match closely with this work: when fitting A, B, and ρ to our data, we measure r2 = 0.99 for all β. The best fit ρ was close to the theoretical value for some values of β (true ρ: 2, best fit ρ: 2.19 for β = 5). Interestingly, we do observe that the best fitting ρ tends to increase with β in our model. Notably, the influence of Agent sub-optimality was outside the scope of this prior work. We are able to observe this relationship between ρ and β through the modelling complexity enabled by our RIRL framework.
sections.heading: 5.2 PRINCIPAL-AGENT IN THE SEQUENTIAL MULTI-AGENT SETTING
sections.text: We show that RIRL enables modeling sequential Principal-Agent problems with multiple Agents with T > 1 timesteps. We consider horizons [2, 10] and teams of na = 4 Agents. We assume there are K = 5 possible Agent abilities k. The Principal cannot see the Agents’ abilities. Each Agent’s ability is sampled randomly at the start of each episode. At each timestep, Agent i’s output is:
  zi,t = hi,t · (νki + ei,t), (9) where it works hi,t hours and exerts effort ei,t. The Principal moves first and sets a wage wi,t. Each Agent moves second: it knows wi,t before choosing hi,t and ei,t, and earning income wi,t · hi,t. As before, Principal utility Up measures profit. We define Ua following standard utility functions, where the optimal h increases withw. As a consequence of this configuration (demonstrated below), the profit-maximizing wage wi for Agent i increases with its ability νki . The Agent utility is:
  Ua(w, h, e) = CRRA(w · h; ρ)︸ ︷︷ ︸ Income Utility − clhα(1 + e)︸ ︷︷ ︸ Work Disutility , Up(w,h, z)︸ ︷︷ ︸ Profit
  = ∑
  i∈[na] zi︸ ︷︷ ︸ Revenue
  − ∑
  i∈[na] wihi︸ ︷︷ ︸ Wages Paid , (10)
  where ρ, cl and α are constants governing the shape of Ua.
  Attention Costs. Because a strategic Principal must infer private Agent features, e.g., ability, its equilibrium behavior depends on any inference costs it experiences, e.g., attention costs. Note that, unlike in the above bandit setting, here we train both the Principal and the Agent policies, each of which is modeled using the RIRL-actor architecture (Section 4). However to isolate and explore the effects of distinct Principal attention costs, we do not impose any attention costs on the Agents. Therefore, the Agents’ reward is only their utility ri,t = Ua(wi,t, hi,t, ei,t).
  For the Principal, we modelM = 3 information channels: one is “easy” and low-cost to observe and two are “hard” and high-cost. The low-cost channel ofp includes information that we regard as freely available (λf = 0), e.g., the time t, the hours worked h (workers often fill out timesheets which makes hi easy to see), and the total output, Z = ∑ i∈[na] zi (managers can see the final result). However, it is high-cost to see individual contributions: we use a high-cost channel oep for efforts e, and a high-cost channel ozp for outputs z. This models a Principal who can spend time and attention to observe individual Agents to reduce uncertainty about their true ability, e.g., their working styles and productivity (Figure 3). Modeling the attention cost of output and effort separately lets us study the effects and interactions of unequal observation costs. The Principal’s reward is
  r†p,t = Up(wt,ht, zt)− λz Ĩ(yzt ; zt)︸ ︷︷ ︸ Individual Output and − λeĨ(yet ; et)︸ ︷︷ ︸ Effort Perception Cost
  (11)
  2Their Principal pays a cost to reduce the noise added to its observed output, as opposed to our MI cost; see the Appendix for full details.
  Thus, the Principal’s bounded rationality is modeled through the cost to get information about effort and individual outputs. The Principal’s RIRL-actor architecture would also let us use attention costs Ĩ(yft ; o f t ) and Ĩ(wt; yt), but we omit those here 3. Additional training details are in the Appendix.
  Results. We highlight several noteworthy observations revealed by our RIRL framework: (1) Principal inattention to individual outputs z and inattention to efforts e have distinct, nearly opposite, consequences on actor utilities; (2) Agents respond to the incentive structures created by Principal inattention to effort, with parallels to a prisoner’s dilemma; (3) The temporal patterns of Principal attention reflect the value of information over time.
  Welfare Implications of Modeling Bounded Rationality. We analyze welfare related to actors’ utilities. At equilibrium, the Principal and Agent utilities are negatively correlated (Figure 4a,b), comparing across varying levels of λz and λe. Note that the “rational” model has λz = λe = 0. This indicates that the Principal’s bounded rationality has opposing implications for the Principal and Agents. From Figure 4a, we see that Agents’ average utility increases and the Principal’s utility decreases when the Principal’s attention cost for individual outputs λz increases. Conversely, Agent utility decreases with increasing Principal attention cost on effort λe.
  The cause for this is as follows. The Principal has a different optimal wage for each ability νk as shown in Figure 4c. When the Principal is fully rational it can use output and effort to accurately infer ability. Similar to what was seen in the bandit setting (above), increasing attention costs (in this case, on output λz) lead the Principal to set wages in a manner that is less profitable but also less attentionally demanding. At the resulting equilibria, the Principal has more uncertainty over each Agent’s type and adopts a “better safe than sorry approach” and increases the average wage to ensure output. In sum, the same force that creates a lower-utility equilibrium for the Principal has higher average utility for the Agents.
  3We do add entropy regularization over ω(a|y) for both the Principal and Agent to encourage exploration.
  However, while the average Agent utility increases with λz , it does not increase for all Agent types. Specifically, the utility of the (highest) lowest-ability Agent’s (decreases) increases. Hence, the Principal’s uncertainty over individual outputs decreases the wage (and utility) differences between Agents of different ability (Fig. 4c,d). This is particularly relevant when considering the equality of utility. A common inequality metric is the Gini coefficient (Gini, 1971), computed as a normalized sum of income differences; equality can be defined as eq = 1 − NN−1gini. Figure 5b shows that Agents’ average utility and equality across types both increase for higher output attention cost λz .
  Bounded Rationality induces Social Dilemma Dynamics. This setup gives rise to interesting temporal strategic behaviors akin to signalling (Spence, 1973). Note that, while effort e increases an Agent’s work disutility without increasing its income, Agents may still choose to exert non-zero effort e > 0. To provide an intuition for this, first, note that spending effort increases output rate (z per hour h), resembling higher ability νk. Second, the profit-maximizing wage wi increases with Agent ability4 νki . The Principal can’t see each ν k i but can see the output zi and hours hi. Hence, output rate is a “signal” of ability and Agents can misrepresent their ability by spending effort.
  When λe increases, it becomes costly for the Principal to distinguish between the Agent’s ability and effort-action. Interestingly, this leads to signaling equilibria in which Agents choose to use more effort (Fig. 4a), resulting in lower average Agent utility and higher Principal utility (Fig. 5a). In effect, this Agent behavior has parallels to the “defect-defect” equilibrium in the prisoner’s dilemma (Shoham & Leyton-Brown, 2008). When higher-ability Agents are not using effort, lowerability Agents are incentivized to use effort to misrepresent themselves as having higher ability. This incentivizes the higher-ability Agents to also use effort to distinguish themselves from the lower-ability Agents. Figure 5a shows this effect: effort increases with λe at all ability levels. In effect, increasing λe creates equilibria where the Principal enjoys higher Agent effort essentially for free. Interestingly, the effects of effort being costly to observe are nullified if the individual outputs are also hard to observe (Fig. 5b), showing these two cost parameters interact.
  RIRL-actors Learn the Time-Value of Information. We also show that RIRL can discover the time-value of information-acquisition and how it depends on the time horizon. Figure 5c shows output information that the Principal encodes Ĩfz (yzt ; zt) over time. There is an initial spike in the amount of information at the beginning of the episode and this decreases over time. Intuitively, it is most efficient for the Principal to pay attention to outputs at the beginning of the episode, as this information can be used throughout the episode. Additionally, the initial spike is larger for longer horizons, as initial information has more value for longer episodes.
  Limitations. While RI is a general model of boundedly rationality, it does not cover all models of human irrationality. For simulations that wish to use more specialized models of bounded rationality, RIRL may not be sufficient. Examples include hyperbolic discounting (Kirby & Herrnstein, 1995) to model time-inconsistent delay discounting (humans tend to prefer rewards in the near future much more than rewards farther in the future) or prospect theory (Tversky & Kahneman, 1992) to model loss-aversion. Future research may extend RIRL to include such notions of bounded rationality.
sections.heading: 6 CONCLUSION
sections.text: We propose a novel framework, RIRL, and associated policy class for modeling bounded-rationality in MARL simulations with complexity beyond the scope of prior techniques. Our method incorporates Rational Inattention, an established model of human bounded rationality which uses mutual information to model the cognitive cost of processing information. Mutual information has been used extensively in RL for exploration and skill discovery, but, to our knowledge, we are the first to incorporate it for modeling human-like behavior in multi-agent simulations. We evaluate our method in two Principal-Agent problem settings, including a complex multi-agent setting, with multiple information channels with heterogeneous costs. Incorporating bounded rationality leads to different actor strategies and welfare outcomes as compared to under rational assumptions. These results establish RIRL as a promising framework for using MARL to analyze systems of human agents.
  4This is a simple consequence of how utility functions are defined. See Fig 4c for confirmation.
sections.heading: 7 REPRODUCIBILITY AND ETHICS STATEMENTS
sections.text: Ethics Statement. Our work proposes a framework to model bounded rationality in Multi-Agent Reinforcement Learning based simulations. Thus our framework may be used to draw implications in simulations modeled after real-world systems. While addressing the rationality gap between human actors and RL actors is an important step towards this goal, it is hardly sufficient for achieving the realism required for real-world decision making based on AI simulations. We anticipate a much longer path forward before this technology matures to such a degree and note that any premature usage of our framework towards real-world decision making would contradict its intended research purpose. Furthermore, our framework should not be used to explore methods to increase discrimination or unfairness in real-world systems, instead it should be use to investigate how to decrease such biases. We acknowledge that our PA settings identify a tension between employer (Principal) profit and employee (Agent) utility/equality. Our results are not meant to provide any actionable insight into how this tension could be manipulated for the benefit of one party. Any such application of our results or our proposed framework would constitute an ethical violation.
  Reproducibility Statement. We discuss specific training details for all experiments such as hyperparameters, random seeds used, and our process for setting the random seed in the Appendix. Upon acceptance, all code for this project will be made open source and publicly available for reproducibility purposes and further research. We will share our code with reviewers and ACs for review upon the opening of the discussion forums.
sections.heading: A ADDITIONAL BANDIT EXPERIMENT DETAILS AND RESULTS
sections.text: A.1 ADDITIONAL DETAILS
  Noise Structure. Recall in the bandit experiments, the output z is a noisy function of the hours action h chosen by agent. Specifically we use the following output distribution:
  p(z|h) = { 0.7 if z = h 0.7|z−h|
  0.3·c otherwise (12)
  Where c is a numerically calculated factor such that p(z|h) is a proper probability distribution satisfying ∑ z∈Z p(z|h) = 1. This noise distribution assures high correlation between h and z with the probability of z levels decreasing exponentially as the difference between h and z grows. We plot the probability heatmap in Figure 6a.
  Constant Relative Risk Aversion (CRRA) Function The constant relative risk aversion (crra function (Pratt, 1978) commonly used in economics has constant risk aversion which means decisions are invariant to scale. It is a concave function where the risk aversion parameter, ρ ≥ 0, determines the concavity. This models diminishing returns with higher values. The function is given by:
  CRRA(u, ρ) =
  { u(1−ρ)−1
  1−ρ if ρ ̸= 1, ρ ≥ 0 ln(u) ρ = 1
  (13)
  Inattention Model From Theory. The model studied in classical theoretical Principal-Agent literature we compare with (Mirrlees, 1976) utilizes the same utility functions for the principal and the agent. It has a slightly different model of principal inattention where it models costly principal inattention with a factor θ that controls the magnitude of the observation noise in the output signal. Specifically, for a true output level z∗, the principal observes output z = z∗ + 1θ ϵ where ϵ is noise. The principal incurs an attention cost that is a function of θ2 and is subtracted from their utility function. While this attention model is different from our mutual information based model, it shares many similar properties. In both our models the inattention introduces noise which makes acting optimally difficult. Additionally in both models, an attention parameter can reduce this noise at a cost to the principal and the principal can choose the optimal, utility-maximizing value of this attention parameter. Such similarities explain why, even under different models, our results are similar to those from theoretical analysis.
  Training Hyperparameters. We used learning rates of 1e-3 for the training the principal policy parameters and 5e-3 for the mutual information classifier. We used a batch size of 128 and trained the principal for a total of 100000 batches. During training we gradually annealed λap from 0 to the desired value at a rate of 4/10000 per batch. We average all results across 5 random seeds and we set the random seed for pytorch, numpy, and python’s internal random module for each run. We used seeds of [0, 4]. All experiments were run on 16CPU cloud compute machines with 54GB of memory. Given a pay schedule which consists of mean and standard deviation parameters (µz,σz) for each output level z ∈ Z , to calculate the principal policy we first sample 100 pay schedules and calculate the agent utility per output for each output level for each pay schedule. We then average to
  calculate the average agent utility for each output level. We use the noise structure to calculate the average utility for each action and use the soft-q formulation over the utilities given in the main text to obtain the agent’s stochastic policy.
  A.2 ADDITIONAL RESULTS
  We provide a few additional figures for the bandit experiment. Recall the principal’s policy was the pay schedule parameterized by mean and standard deviation values for each output level (µz,σz). Figure 6b shows the standard deviation parameters σz learned under MI and Entropy regularization for different cost factors λ. Increasing entropy regularization results in very large increases to the standard deviation parameters while MI regularization leads to much smaller standard deviation increases. The pay schedule means for Entropy regularization become sharper to overcome this increase in standard deviation.
  Figure 6c compares pay schedule means (µz) across different Agent policy temperature (β) values. For low inattention costs, the pay schedules are similar. However the pay schedules for higher β (more deterministic agents) decrease slower with increasing principal attention cost. Given the pay schedule parameters, the agent has an optimal h action. More deterministic agents take this optimal action with higher probability so it is more valuable for the principal to incentivize them to higher optimal actions, which in turn lead to higher outputs. This explains why Agent and Principal utility are higher for higher β values.
sections.heading: B ADDITIONAL MULTI-AGENT MULTI-TIMESTEP EXPERIMENT DETAILS AND RESULTS
sections.text: B.1 ADDITIONAL DETAILS
  Training Hyperparameters. We used learning rates of 1e-4 for the Principal and Agent’s policy parameters and 1e-3 for all the mutual information classifiers. We used a batch size of 512 episodes and train the principal and agent through 60000 batches. We train a single RIRL-actor for the
  Agents and concatenate the experiences of all na Agents when updating the policy. The Agent policy therefore effectively has a batch size of 512na. To avoid vastly different total episode returns, we scaled the rewards by the horizon during training. We run all experiments on 8CPU cloud computing machines with 26GB of memory. We average all results across 20 random seeds and we set the random seed for pytorch, numpy, and python’s internal random module for each run. We used seeds of [0, 19].
  B.2 ADDITIONAL RESULTS
  We present some additional results for the sequential multi-agent experiments. Figures 7a,7b show Principal utility with output MI cost (λz) and effort MI cost (λe) for different horizons. We see the trend of Principal utility decreasing with λz and increasing with λe occurs across all horizons.
  Figure 7c shows unscaled MI cost for output and effort with λz and λe. We see that I(ye|e) decreases with both λz and λe. I(yz|z) decreases with λz but is not affected greatly by λe. Figure 7d plots the agent utility for all agent types. It expands the plot of Figure 4d. The plots show the difference between the utility for each agent type and the average utility.
  Figure 7e demonstrates how different values for different agent types change with output MI cost (λz). We plot each result as the change from the λz = 0 values. As mentioned previously, utility
  increases with λz for lower ability agents while it decreases for the highest ability agent. We observe similar trends for hours h. We additionally notice effort e generally decreases which leads to z staying constant or decreasing for all agent types, even though some agent types have increased hours. This lower output along with higher average wages explains the decrease in Principal utility.
sections.heading: C RIRL IMPLEMENTATION DETAILS
sections.text: We use this section of the appendix to cover important implementation details and tips. This is intended for practical guidance.
  Stochastic Encoder Module Configuration and Initialization. Section 4 describes the architecture of our RIRL-actor policy class. As described, we learn an encoder fm(ymt |omt , ψt) for each observation channel m. Encoder fm takes observation omt and recurrent state ψt as inputs and outputs the parameters (means and standard deviations) of a stochastic encoding ymt .
  We find that, in practice, learning does not progress if each ym contains very little information about om at the start of training. To address this, we recommend two implementation choices. First, implement fm as a residual-style module:
  µmt , σ m t = f m(omt , ψt), y m t = o m t + µ m t + σ m t · ϵmt , (14)
  This simply requires setting the output ym to have the same size as observation om and adding omt to the mean µmt . Second, initialize the output layer of f
  m such that σm is consistently very small. We perform this by adding a constant negative offset to the bias units associated with the log σm outputs, which we exponentiate to get σm. As a result of this strategy, ymt closely follows o m t at the start of training.
  Hidden State as an Encoder Input. We emphasize that the inputs to encoder fm is the concatenation of the observation omt and the hidden state ψt. Similarly, when using the discriminator dm(ymt , [o m t , ψt]) to estimate Ĩfm and when training the discriminator, we also apply this concatenation. In other words, dm regards [omt , ψt] as the observation, such that the Ĩfm captures the MI between ymt and [o m t , ψt].
  This is an important detail for ensuring that MI regularization works as expected in the multi-step setting. For instance, we observed that, if the discriminator does not see the hidden state ψ, fm learns to encode om such that I(ym; om) is minimal but where omt can still be easily recovered given ymt and ψt.
  Optimization During training, we found that learning was most stable if separate learning rates were used for the policy modules {f1, . . . , fM ,LSTM, ω} and for the discriminator modules {d1, . . . , dM , dω}. Importantly, the discriminator modules use a 10x higher learning rate. Concretely, we use learning rates of 0.0001 and 0.001 for the policy and discriminator modules, respectively. Configuring learning rates this way helps to ensure that discriminator dm can adjust to changes in encoder fm faster than the encoder can adapt to changes in the discriminator. Intuitively, this improves the quality of the MI estimates during training.
  Another important optimization detail concerns gradient flow. Gradients from ∇ logω(at|·) need to backpropagate through the encodings [ y1t , . . . y M t ] in order for the encoder modules to receive meaningful gradients. In Pytorch, which we use for this implementation, ensuring that this gradient flow occurs requires some attention. Given the (learned) mean and standard deviation parameters (which are functions of the encoder input), Pytorch constructs the sampling distribution as m = Normal(µ, σ). The output of the encoder module ymt must be sampled via the reparameterization trick: y = m.rsample(), which allows gradients to flow through ymt . Finally, this output should be detached from the backpropagation graph when calculating its log probability: encoder log prob = m.log prob( y.detach() ), which is needed for computing the policy gradients. Similarly, care must be taken to ensure that inputs to the discriminators are detached in the same way.