summaries: The paper investigates emergent gesture-based communication in Embodied Multi-Agent Populations. A noticeable feature of the paper is that it investigates emergent communication in the case of non-uniform distribution of intents and costly communication (i.e. agents are penalized for effort). The authors find that in certain scenarios, these conditions may lead to communication generalization of learned communication strategies to new, previously unseen agents.

summaries: The paper deals with agents that communicate non-verbally via actuating their joints in a 3D environment. The authors show that the agents should be able to learn protocols that can generalize to novel partners. Furthermore, the authors find that the current training approaches are brittle, and they propose and evaluate approaches to address this challenge.

summaries: The authors study the zero-shot emergent non-verbal communication in this paper. Different from most papers on emergent communication. this paper uses the motion of three-joint agents. The agents meet partners that they have never seen in the training phase, presenting the challenge of the universal protocol. To make a universal protocol possible, the authors study intents sampled from Zipf distribution and energy regulation. The authors conducted experiments on tasks with 2, 5, 10 intents. The results show that providing latent energy feature is essential for zero-shot coordination. To achieve better than chance accuracy on tasks with 10 intents, a torque curriculum is needed.

abstractText: Effective communication is an important skill for enabling information exchange and cooperation in multi-agent settings. Indeed, emergent communication is now a vibrant field of research, with common settings involving discrete cheap-talk channels. One limitation of this setting is that it does not allow for the emergent protocols to generalize beyond the training partners. Furthermore, so far emergent communication has primarily focused on the use of symbolic channels. In this work, we extend this line of work to a new modality, by studying agents that learn to communicate via actuating their joints in a 3D environment. We show that under realistic assumptions, a non-uniform distribution of intents and a commonknowledge energy cost, these agents can find protocols that generalize to novel partners. We also explore and analyze specific difficulties associated with finding these solutions in practice. Finally, we propose and evaluate initial training improvements to address these challenges, involving both specific training curricula and providing the latent feature that can be coordinated on during training.

authors:
Daniel S Bernstein
Robert Givan
Neil Immerman
Shlomo Zilberstein

title: The complexity of decentralized control of markov decision processes

venue: Mathematics of operations research,

authors:
Diane Bouchacourt
Marco Baroni

title: How agents see things: On visual representations in an emergent language game

venue: In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing,

authors:
Kris Cao
Angeliki Lazaridou
Marc Lanctot
Joel Z Leibo
Karl Tuyls
Stephen Clark

title: Emergent communication through negotiation

venue: In International Conference on Learning Representations,

authors:
Rahma Chaabouni
Eugene Kharitonov
Emmanuel Dupoux
Marco Baroni

title: Anti-efficient encoding in emergent communication

venue: In Advances in Neural Information Processing Systems,

authors:
Tom Eccles
Yoram Bachrach
Guy Lever
Angeliki Lazaridou
Thore Graepel

title: Biases for emergent communication in multi-agent reinforcement learning

authors:
Katrina Evtimova
Andrew Drozdov
Douwe Kiela
Kyunghyun Cho

title: Emergent communication in a multi-modal, multi-step referential game

authors:
Jakob Foerster
Ioannis Alexandros Assael
Nando de Freitas
Shimon Whiteson

title: Learning to communicate with deep multi-agent reinforcement learning

authors:
Jakob Foerster
Francis Song
Edward Hughes
Neil Burch
Iain Dunning
Shimon Whiteson
Matthew Botvinick
Michael Bowling

title: Bayesian action decoder for deep multi-agent reinforcement learning

venue: In International Conference on Machine Learning,

authors:
Deepak Gopinath
Jungdam Won

title: Fairmotion - tools to load, process and visualize motion capture data

venue: Github,

authors:
Laura Harding Graesser
Kyunghyun Cho
Douwe Kiela

title: Emergent linguistic phenomena in multi-agent communication games

venue: In Proceedings of the Conference on Empirical Methods in Natural Language Processing and the International Joint Conference on Natural Language Processing (EMNLP-IJCNLP),

authors:
Serhii Havrylov
Ivan Titov

title: Emergence of language with multi-agent games: Learning to communicate with sequences of symbols

authors:
Nicolas Heess
Gregory Wayne
David Silver
Timothy Lillicrap
Tom Erez
Yuval Tassa

title: Learning continuous control policies by stochastic value gradients

authors:
Daniel Holden
Taku Komura
Jun Saito

title: Phase-functioned neural networks for character control

venue: ACM Transactions on Graphics (TOG),

authors:
Hengyuan Hu
Adam Lerer
Alex Peysakhovich
Jakob Foerster

title: Other-play for zero-shot coordination

authors:
Satwik Kottur
José Moura
Stefan Lee
Dhruv Batra

title: Natural language does not emerge ‘naturally’in multi-agent dialog

venue: In Proceedings of the Conference on Empirical Methods in Natural Language Processing,

authors:
Marc Lanctot
Vinicius Zambaldi
Audrunas Gruslys
Angeliki Lazaridou
Karl Tuyls
Julien Pérolat
David Silver
Thore Graepel

title: A unified game-theoretic approach to multiagent reinforcement learning

authors:
Angeliki Lazaridou
Alexander Peysakhovich
Marco Baroni

title: Multi-agent cooperation and the emergence of (natural) language

venue: arXiv preprint arXiv:1612.07182,

authors:
Angeliki Lazaridou
Karl Moritz Hermann
Karl Tuyls
Stephen Clark

title: Emergence of linguistic communication from referential games with symbolic and pixel input

authors:
Ryan Lowe
Jakob Foerster
Y-Lan Boureau
Joelle Pineau
Yann Dauphin

title: On the pitfalls of measuring emergent communication

venue: In Proceedings of the 18th International Conference on Autonomous Agents and MultiAgent Systems,

authors:
Ryan Lowe
Abhinav Gupta
Jakob Foerster
Douwe Kiela
Joelle Pineau

title: On the interaction between supervision and self-play in emergent communication

authors:
Igor Mordatch
Pieter Abbeel

title: Emergence of grounded compositional language in multi-agent populations

venue: In Thirty-Second AAAI Conference on Artificial Intelligence,

authors:
Tabish Rashid
Mikayel Samvelyan
Christian Schroeder
Gregory Farquhar
Jakob Foerster
Shimon Whiteson

title: Qmix: Monotonic value function factorisation for deep multi-agent reinforcement learning

title: The evolution of grounded spatial language

authors:
Michael Spranger
Luc Steels

title: Emergent functional grammar for space

venue: Experiments in Cultural Language Evolution,

title: Language games for autonomous robots

venue: IEEE Intelligent systems,

authors:
Luc Steels
Michael Spranger

title: Emergent mirror systems for body language

authors:
Luc Steels
Michael Spranger
Remi Van Trijp
Sebastian Höfer
Manfred Hild

title: Emergent action language on real robots

venue: In Language grounding in robots,

authors:
Richard S Sutton
David A McAllester
Satinder P Singh
Yishay Mansour

title: Policy gradient methods for reinforcement learning with function approximation

title: Simple statistical gradient-following algorithms for connectionist reinforcement learning

venue: Machine learning,

title: Human behavior and the principle of least effort: An introduction to human ecology

venue: Ravenio Books,

heading: 1 INTRODUCTION

text:
The ability to communicate effectively with other agents is part of a necessary skill repertoire of intelligent agents and, by definition, can only be studied in multi-agent contexts. Over the last few years, a number of papers have studied emergent communication in multi-agent settings (Lazaridou et al., 2016; Havrylov & Titov, 2017; Cao et al., 2018; Bouchacourt & Baroni, 2018; Eccles et al., 2019; Graesser et al., 2019; Chaabouni et al., 2019; Lowe et al., 2019b). This work typically assumes a symbolic (discrete) cheap-talk channel, through which agents can send messages that have no impact on the reward function or transition dynamics. A common task is the so called referential game, in which a sender observes an intent needing to be communicated to a listener via a message.
In these cheap-talk settings, the solution space typically contains many equivalent but mutually incompatible (self-play) policies. For example, permuting bits in the channel and adapting the receiver policy accordingly would preserve payouts, but differently permuted senders and receivers are mutually incompatible. This makes it difficult for independently trained agents to utilize the cheap-talk channel at test time, a setting which is formalized as zero-shot (ZS) coordination (Hu et al., 2020).
In contrast, we study how gesture-based communication can emerge under realistic assumptions. Specifically, this work considers emergent communication in the context of embodied agents that learn to communicate through actuating and observing their joints in simulated physical environments. In other words, our setup is a referential game, where each message is a multi-step process that produces an entire trajectory of limb motion (continuous actions) in a simulated 3D world.
Not only does body language play a crucial role in social interactions, but furthermore, zoomorphic agents, robotic manipulators, and prelingual infants are generally not expected to use symbolic language to communicate at all. From a practical point of view, it is clear that our future AI agents will need to signal and interpret the body language of other (human) agents, e.g., when self-driving cars decide whether it is safe to cross an intersection.
With that, there has been work on the emergence of grounded language for robots (Steels et al., 2012; Spranger, 2016). To the best of our knowledge however, we are first to explore deep reinforcement learning for emergent communication in the context of embodied agents using articulated motion.
Moreover, while cheap-talk is a great proxy for symbolic communication across dedicated channels, communication through articulated motion means agents have to control their joints to signal. One universal feature of this physical actuation is it requires the expenditure of energy, which is a scarce resource both for biological agents and man-made robots.
Another ubiquitous factor of the physical world (and many other domains) is that communicative intents are not distributed uniformly. In particular, the Zipf distribution (Zipf, 2016) is known to be a good proxy for many real-world distributions associated with human activity.
A consequence of combining energy cost with a non-uniform distribution over intents in the
context of referential games is that, in principle, it allows for ZS communication: Trajectories requiring lower energy exertion should be used for encoding more common intents, while those associated with higher energy encode less common ones.
In contrast, superficially related, auxiliary losses such as entropy penalties do not allow for ZS coordination without further assumptions. While these auxillary losses are design decisions, energy cost is an example of a universal (common-knowledge) cost grounded in the environment, which can be exploited for ZS communication.
Unfortunately, training agents that can successfully learn these strategies is a difficult problem for current state-of-the-art machine learning models. There are three major challenges: 1) Local optima associated with the lock-in between sender and receiver: The interpretation of a message depends on the entire policy, not just the state and action. While recent methods have been developed to address this in discrete action spaces (Foerster et al., 2019), to the best of our knowledge none have been proposed for costly, continuous action spaces. 2) The latent structure underlying the protocol, in our case energy, that can be coordinated on, needs to be discovered, requiring agents to ignore other (redundant) degrees of freedom. 3) Even when this structure is provided, optimization is difficult since it contains a large number of local optima. As a consequence, on top of the continuous optimization problem there is a combinatorial problem of ordering the energy values for each intent.
We explore and analyse these difficulties, suggesting initial steps for addressing them in our setting. First, we show that providing the latent variable (energy) at training time does indeed allow for some amount of ZS coordination. To do so, we adapt our method two fold: 1) During training, we change the observation to only the energy value of a given trajectory. 2) We add an observer agent that trains on an entire population of fully trained Self-Play (SP) agents. To evaluate ZS performance, we test this observer on an independently trained set of SP agents. The ZS performance in this setting is around 35% for 10 intents, comparable with the most frequent class for the Zipf-distribution (34%).
Next, we pretrain sender agents to minimize the energy associated with each intent, but ZS performance remains around 34% (10 intents). This is intuitive due to the challenges associated with re-ordering energy values on a 1D line without incurring a high cross-entropy while the different distributions are overlapping. Finally, we show that using the entire trajectory during SP, but only the energy value for the external observer, in combination with pretraining, consistently achieves a much higher ZS performance of around 56%.
All of this illustrates that learning an optimal ZS policy given a specific set of assumptions about the problem setting, which are common knowledge to all parties, is a challenging problem, even in seemingly simple instances. Furthermore, since this work focuses on the two extreme ends of this problem (self-play and ZS), our ideas are relevant for a broad range of intermediate settings as well.

heading: 2 BACKGROUND

heading: 2.1 MULTI-AGENT REINFORCEMENT LEARNING

text:
We formalize the protocol learning problem as a decentralized partially observable Markov decision process with N agents (Bernstein et al., 2002), defined by tuple (S,A,T,R,O,Ω,γ). S is the set of states, A1 · · ·AN the set of actions for each of N agents in the population, and T a transition function S×A1 · · ·AN→ S mapping each state and set of agent actions taken to a distribution over next states. This work assumes all agents have identical state and action spaces. In a partially observable setting, no agent can directly observe the underlying state s, but each receives a private observation oi ∈ Ω correlated with the state. O(o|a,s′) is the probability that o is observed, given action a led to state s′. Agent reward ri : Ω×A1 · · ·AN→R is a function of state and actions taken. The objective is to infer a set of agent policies that maximize expected shared return R. We employ a centralized training regime with execution decentralized (Foerster et al., 2016; Lanctot et al., 2017; Rashid et al., 2018).
Policy gradient algorithms (Williams, 1992; Sutton et al., 2000) are widely used for reinforcement learning domains with continuous action spaces. Similar to (Mordatch & Abbeel, 2018), we use a model-based policy gradient approach with a fully differentiable dynamics model for our emergent communication framework. In particular, we employ a stochastic value gradient approach, SVGinfinity (Heess et al., 2015). SVG methods compute the policy gradient through backpropagation.

heading: 2.2 ZERO-SHOT COORDINATION

text: Zero-shot (ZS) coordination is the problem setting of agents coordinating at test time with novel partners, i.e. other independently trained agents (Hu et al., 2020). In cooperative multiagent settings, a key challenge is for agents to learn general skills for coordinating and communicating with other agents. Nonetheless, just as in single-agent settings, agents can overfit to their environment, in multiagent settings, agents can co-adapt with and overfit to their training partners. Thus ZS coordination is useful for evaluating how well the learned agent policies generalize to unseen agents they may need to later coordinate with (e.g. new human partners).

heading: 3 PROBLEM SETTING

text:
This work is framed in the ZS Communication problem setting. ZS communication is a specific instance of the ZS coordination setting, where the task for the agents is to communicate effectively; there is no additional cooperative task being solved concurrently, in the environment. The protocol learning problem then for all independently trained sender-receiver agent pairs is to converge upon optimal zero-shot policies, for communicating intent to novel partners at test time. Sender-receiver pairs engage in self-play to train communication protocols, then we use cross-play as our evaluation metric for testing ZS coordination. Cross-play (CP) is an instance of out-of-distribution testing, whereby senders are paired with receivers they have never encountered nor influenced during training (i.e. agent pairs tested in CP have been trained independently of one another).
In our embodied problem formulation, populations are composed of spatially articulated agents. Concepts represent communicative intents and messages are instantiated as motion trajectories. Intents are represented as a discrete-valued symbols, defined a priori from an intent library. The policy state space S consists of the joint configuration of an agent and an intent to be communicated, where the joint configuration of the agent is defined by three-dimensional position (px, py, pz) and threedimensional rotation (rx,ry,rz) of each agent joint j in the agent’s set of joints J. The policy action space A consists of the angular velocities for each joint j ∈ J: (∆rx,∆ry,∆rz). All a ∈ A are communication actions which can be observed by other agents but have no effect on the environment.
We employ referential games for generating communicative motion through paired-play. At each time step t, the policy inputs the joint configuration of the agent, (px, py, pz,rx,ry,rz) ∀ j ∈ J, concatenated with a fixed intent embedding. It outputs an action at , (∆rx,∆ry,∆rz) ∀ j ∈ J. Transitions T : (st ,at)→ st+1 are deterministically computed through a differentiable forward kinematics (FK) module, ensuring kinematically valid trajectories are generated. After T steps, the episode terminates, and the sequence of joint states visited and velocity actions taken is concatenated to compose a motion trajectory. The observer model takes the motion trajectory as input and predicts the actor’s intent. The shared return for the episode is the cross-entropy loss between the observer’s prediction
and the ground truth intent; this return R is backpropagated from observer to actor, since we use a centralized training regime for the multi-agent system. The FK module has been adapted from an existing motion library (Holden et al., 2017) to be differentiable. Figure 1 illustrates the embodied referential game. Algorithm 1 provides an overview of training for the game.
Importantly, for continuous domains, the injection of noise is critical. During trajectory generation, Gaussian noise is added to each action degree of freedom, both to represent imperfect observability and to induce exploration. Noise helps regularize the communication channel, in that a finite noise level essentially provides a quantization of the continuous space, a ’coarse graining’, such that trajectories mapped to different intents are distinguishable. This prevents a collapse of the trajectories.

heading: 4 METHODOLOGY

text:
Given that we aim to develop agents with general communication skills (policies that are not simply locally optimal to their current communication partner), it is useful to first consider why converging on an embodied protocol that generalizes to novel partners is a challenging problem.
Since embodied communication is physically instantiated, it occurs through manipulating the velocities of agent joints; thus, communication through motion generation implies a high-dimensional continuous action space. Combining the training criterion with such a high dimensional action space results in a highly non-convex optimization surface. This means the optimization landscape has many local optima (reasonable solutions) for inferring a protocol. It can be difficult for a local optimization algorithm to navigate this type of landscape in search of a global optimum. It would also be sensitive to where in the policy space policy parameters are initialized. Because of these challenges, our approach proposes to induce latent structure during protocol training, to provide similar grounding for how independent actors generate communication. This structure can subsequently be exploited to improve ZS coordination at test time.

heading: 4.1 INDUCING IMPLICIT LATENT STRUCTURE THROUGH PHYSICAL ENERGY EXERTION

text:
To implicitly induce latent structure in the policy learning process (and thus trajectory generation), we propose to use Energy Regularization coupled with a Zipf distribution over intents. The regularizer enforces minimal energy trajectories for the protocol, and Zipf imposes a monotonic ordering upon intents, based upon likelihood. Coupled, they incentivize an inverse relationship between energy exertion and intent frequency, assigning minimum energy trajectories to maximally occurring intents. Moreover in principle, there exists a global optimum for the protocol: energy values can be ordered to be strictly increasing, and intents can be ordered to be strictly decreasing. Then presumably, a 1:1 mapping between energy values and intents can be induced.
In practice however, this is a very challenging optimization problem, primarily because the optimization surface has many local optima for inferring a protocol. Given the latent structure, there is additionally the problem of strictly ordering energy values, which is combinatorial in the number of intents. Different local optima will solve this problem in different ways, and because energy values are continuous, reasonable solutions can be obtained (and achieve low loss) without matching the correct ordering. This is especially true as the number of concepts to be learned grows and increasing numbers of intents have very similar likelihoods of occurring. Additionally, because the learning problem for each agent is to infer one policy that effectively communicates multiple intents, if the optimization algorithm gets stuck in a local optimum, finding better local optima likely requires non-local traversal through the policy space. This is because the algorithm may need to change the actions for multiple intents concurrently, to locate a better solution in the space.
For inducing implicit structure, there are two objectives traded off: maximization of communication success (Equation 1) and minimization of energy exertion (Equation 2). Let us denote the set of intents (goals) to be communicated G and motion trajectory τ ∈ T , the set of all trajectories. We employ an L2 torque loss where I is the moment of inertia and ω the angular velocity, for all agent joints. The total loss is a linear combination of prediction (cross-entropy) and energy (torque) losses.
Lpred =− log pφ (ĝ = g∗ | τ) (1)
Lengy =‖ I ∗ (ω̂1:T − ω̂0:T−1) ‖22 (2)
The latent structure is implicit because agents are never made aware that any predefined, exploitable structure exists, nor are they directly incentivized to discover a more compact representation. This means if agents fail to autonomously discover the induced structure, they will be relegated to overfitting to trajectory input with their current partners and though they might perhaps find some structure, it will be insufficient for ZS coordination with novel partners. For this reason, in our experiments, we examine two ways of giving input to observer models: (1) provide actor trajectories generated, as this ideally is the goal, that the observer can autonomously discover any latent structure and exploit it for decoding messages, and (2) directly provide the latent energy values, as this allows decoupling of successfully induced structure in the policy learning from the successful discovery of that structure for the interpretation of messages. Both are necessary for successful coordination.

heading: 4.2 PROVIDING EXPLICIT LATENT STRUCTURE

text:
For explicitly providing structure, we define a set of latent features on trajectories Φ(T ). This structure is intended to reduce dimensionality of messages passed, while preserving informativeness of messages. Thus we aim to induce a relationship I(G;τ) = I(G;Φ(τ)) 0, to achieve high mutual information between G and Φ. This relationship can be expanded as I(G;Φ) = H(G)−H(G|Φ). However, the intent distribution is given as a Zipfian, so H(G) is held constant. Thus, as is generally true in protocol learning, maximizing mutual information between goals and messages implies:
min H(G|Φ) =⇒ {
p(g|Φ(τ))→ 1 g = g∗ p(g|Φ(τ))→ 0 else (3)
This implication is critical for ZS coordination, in the absence of data to bias how agents learn to generate trajectories. It suggests that even if a new observer does not know how a particular actor communicates, i.e., it has not learned a mapping between actor trajectories and intents, if it can successfully infer the latent structure for encoding trajectories, it can still decode the message.

heading: 4.3 THIRD-PARTY OBSERVER EVALUATION

text: We evaluate policies learned using a third-party (external) observer, not trained on the protocol. Learned policies are frozen and the external observer initialized randomly. The population is split into disjoint training and test sets. At each iteration, it trains by only observing actors in the training set, and tests on its ability to correctly interpret intent of actors in the test set. Since all actors are trained independently, the ability to understand these unseen partners represents ZS communication.

heading: 5 EXPERIMENTS AND RESULTS

text: Our experiments analyze: (1) the value of inducing implicit latent structure on embodied protocol learning, (2) the impact of discovery of the latent structure by observer agents – in the ZS coordination setting, and (3) challenges associated with optimizing in a continuous space for a monotonic ordering on energy values. We also provide some qualitative insights about learned policies. The experimental setup is detailed in the Appendix.

heading: 5.1 EFFICACY OF INDUCING IMPLICIT STRUCTURE IN PROTOCOL LEARNING

text: This first set of experiments in Figure 2 reports communication success during training, on a task with 10 intents. Results for our experimental condition (Energy + Zipf) are compared against all three ablations. Figure 2a shows all conditions successfully converge on a protocol with their training partners (SP), though conditions trading off two objectives converge more slowly. Figure 2b shows communication success for each actor paired with unseen observer agents in the population (all <actor, observer> pairs that did not train together) (CP). This represents an out-of-distribution test case. It shows that none of the models are able to immediately generalize to new partners (at test time), elucidating the need for additional third-party observer training, to increase the likelihood of communication generalization. It also shows there is value in using a nonuniform distribution, as it enables independent agents to agree a priori on how to prioritize intents for future interactions. Lastly, from this experiment, we found that while adding an energy objective does slow convergence, it reduced energy exertion by several orders of magnitude, which can be critical in reducing resource consumption and enabling prolonged operation for practical embodied systems (e.g. robots).

heading: 5.1.1 EFFICACY OF IMPLICIT STRUCTURE IN DISCRETE-CHANNEL PROTOCOLS

text:
To further investigate the value of coupling a Zipfian intent distribution with an energy cost objective for learning ZS policies, we evaluate a simple discrete-channel communication domain. In this simplified domain, we consider protocols with 5 intents and examine two learning tasks. In the first task, there are 10 discrete actions each sender agent can take, each assigned its own unique energy value: Φ(a) = id(a). Examples of learned protocols for this task are visualized in Figure 3. For achieving both maximal prediction accuracy and minimal energy exertion, each intent must be mapped to the lowest energy value it can be assigned, but no two intents can be mapped to the same action. The bottom protocol in Subfigure 3b illustrates a globally optimal solution for this domain in the ZS setting, where discriminability is maximized and energy minimized.
For quantitative analysis on the value of inducing energy costs on top of a Zipfian distribution, we trained five independent sender-receiver pairs, both without an energy penalty and using the penalty to induce the energy-based latent structure. For both conditions, we computed SP and CP performance between and across agent pairs, respectively. While SP performance at the end of protocol training is comparable for the conditions, CP performance is 0.15±0.12 and 0.58±0.22 for the zipf (only) condition vs zipf + energy conditions, respectively. Although this suggests significant variance in performance across pairs for CP evaluation, the key finding is still clear. Additional results for both learning tasks in this domain are discussed in Appendix Sections A.4 and A.5. We also contribute a colab notebook as an instructive tool for exploring protocol learning in this simple domain1. Overall, findings are consistent across tasks: coupling a Zipfian distribution with the energy objective is important for learning more general (ZS) communication skills. We continue our continuous-action protocol experiments using the implicit energy-based latent structure.

heading: 5.2 IMPACT OF LATENT STRUCTURE FOR ZERO-SHOT COORDINATION

text:
While the previous set of experiments highlight the potential value of inducing the proposed implicit latent structure, we now analyze a proof-of-concept task, to extract insights about solving the problem setting of ZS coordination using a high-dimensional continuous communication channel. Table 1 shows ZS results given by the external observer evaluation, for a N = 2 concepts task.
1Colab notebook for experimenting with simple Discrete Domain – http://shorturl.at/luHPX
In the tables, rows represent the type of input given to the observer model at training time and columns represent the type of input given at test time. Inducing a Zipf distribution on the intents, the likelihoods are [0.67,0.33] on concepts 1 and 2, respectively. Thus using a max class classifier as a reasonable baseline for comparison, we would expect a success rate of 0.67. Bold figures highlight conditions that achieved communication generalization, success beyond that of the baseline. The results show near perfect ZS coordination only when providing the external observer with the explicit latent structure (energy values), independent of what information was provided during training.
To gain more insights about why agents are able to generalize so well, given the latent energy variable, we visualize a distribution over energy values for N = 2 intents, for the best performing agent (Figure 4). Only given energy values at test time, the only way for an external observer to perform with such high accuracy is with very little overlap between the intent-conditioned Gaussian distributions. Looking at Figure 4, this is exactly what we see. The left plot shows the condition given trajectory input at training time and the right plot given the latent variable input. The trend is consistent and very clear. Given only these energy values at test time, an external observer can successfully decode the message, independent of the way the specific actor qualitatively communicates through motion. This means the implicitly induced latent structure successfully creates a maximally informative relationship between intent and energy exertion, as characterized by Equation 3.
Figure 5 illustrates snapshots of sample behaviors learned for one agent in the population on the N=2 task. They are visualized using the FAIR Motion Library (Gopinath & Won, 2020). From this visualization, we observe qualitatively distinct communicative behaviors for the two intents.
The primary insight from this proof-of-concept experiment is that we can indeed achieve ZS coordination with this energy-based implicit structure underlying the protocol. However, coordination with novel partners remains improbable unless the latent structure is first discovered by the agents.

heading: 5.3 IMPROVING COMMUNICATION GENERALIZATION WITH A LEARNING CURRICULUM

text:
Observing such high ZS coordination success on the proof-of-concept task is very promising. However, we also aim to characterize performance as the complexity of the communication task increases. With this in mind, we investigated generalization with an external observer on tasks with 5 and 10 communicative intents, respectively. The left side of Table 2 (2a and 2c) shows ZS coordination results after training protocols with the implicit latent structure. The most likely intents are sampled with a likelihood of approximately 0.44 and 0.34 for the N = 5 and N = 10 concepts tasks, respectively; so these probabilities form baselines for comparison against a naive max class classifier. Bold figures highlight conditions with some communication generalization. In the case of the N = 5 task, we see a consistent trend as observed in our proof-of-concept task: the external observer can successfully generalize to unseen actors only when given the explicit latent structure. Though notably, ZS coordination success is substantially diminished with this more difficult communication task. Consistently for all three tasks, when given trajectory input, the external observer cannot outperform simply selecting the most likely intent. Furthermore, with an increase in task complexity to 10 concepts, the external observer never performs better than this baseline strategy.
In analyzing these trends more closely, it is important to consider that as the number of concepts increases, the number of ways to order the corresponding energy values increases combinatorially. It follows that as the complexity of the task increases, the optimization algorithm becomes more susceptible to getting stuck in a local optimum. We conduct analysis on optimizing with the energy objective, to better understand the behavior of the energy optimization and accordingly decide how to address the most salient challenges arising. Figures 6a and 6b show the energy loss over time on the N = 5 concepts task, given both types of input. The plots subsample energy loss values over the
entire duration of training, for each run. Each intent is represented by a different colored line and shaded horizontal lines show the variance of the energy loss, at a particular point in training.
A key observation from this analysis is repeated repositioning of intent ordering as training traverses, and there is noticeably more repositioning of intents when trajectory input is given. This can be explained by the dimensionality of the trajectory input being orders of magnitude larger than the scalar latent input; thus there are many more degrees of freedom in the search space for the optimization algorithm to manipulate the positioning of the intents. Yet for both input types, there is very little repositioning of intent 1. The Zipfian imposes the following approximate distribution over five elements: [0.44,0.22,0.14,0.11,0.09]. Notably, the difference between intent 1 and any other intent is much larger than the difference between any other pair of intents. This suggests that the optimization algorithm would incur too much cost if it orders intent 1 incorrectly, so energy cost of intent 1 generally remains lowest throughout training. By contrast, the positions of much less likely intents (e.g. 4, 5) continue to shift (particularly given trajectory input) and at many points during training, are incorrectly ordered as the optimization algorithm searches for a sufficiently good solution. This implies that the constant shifting of intent-conditioned energy values is bound to impact coordination between actor and observer, since it influences trajectories generated for communication.
From this insight, we decided to pretrain actor policies on energy, by first minimizing torque in trajectory generation for all intents uniformly. Our hypothesis was that training the protocol with an energy-based curriculum may help to reduce the amount of intent repositioning necessary to find the optimal ordering, since the intents will all start with comparable energy exertions. The right side of Table 2 (2b and 2d) shows ZS coordination results after training protocols with the latent structure and torque-based curriculum. The torque curriculum seems to only improve generalization notably when the external observer is given trajectory input for the hardest communication task (N = 10).
Overall, two key insights emerge from our experiments and analyses: (1) it is advantageous to induce an energy-based latent structure for learning generalizable communication protocols and (2) explicit discovery of the latent structure is a separate yet critical piece for ZS coordination to occur.

heading: 6 CONCLUSION AND FUTURE WORK

text: We have presented a first exploration into emergent non-verbal communication in the context of embodied agents in high-dimensional simulated environments. We show that under mild assumptions, the grounding provided by the physical environment should allow our agents to learn protocols that can generalize to novel partners. Yet we also find that the current approaches are more brittle than one might hope. We hypothesize that better robustness could be achieved by methods that can discover the maximum possible coordination from a given level of common knowledge (’grounding’) in the environment. Specifically, this may require distinguishing between those aspects of the optimization problem that are idiosyncratic to a given agent (or might even be shared knowledge) compared to those that are common knowledge (and can therefore be used to coordinate). Overall, while there are many interesting open challenges that remain, this work opens up exciting avenues for exploring continuous action communication protocols in virtually or physically embodied agents.

heading: A APPENDIX

text:
A.1 EMBODIED AGENT MORPHOLOGIES
Embodied agents explored in this work have articulated bodies. An articulated body is collection of rigid bodies (links), connected by a set of joints j ∈ J. The body of an agent is constrained kinematically, such that links cannot detach from one another. Topology defines a tree structure for the body, where each joint is a node and edges denote a link between joints. The root joint, j0, is the root of the tree. Only the root joint has six degrees of freedom (DOF): 〈 px, py, pz,rx,ry,rz 〉 . Since the body must move as one entity and thus links cannot detach from one another, given no further joint limits or constraints, all other joints have the ability only to rotate freely in Euclidean space. So all derivative (child) joints ji where i 6= 0 have three DOFs: 〈 rx,ry,rz 〉 . For simplicity, we constrain all agent joints with only the DOFs to rotate in three-dimensional space. This means agents cannot navigate through the environment; their bodies can only move in place. Moreover, a predefined topological structure implies trajectories generated must adhere to the kinematic constraints of the agent given a priori. We employ a forward kinematics (FK) transition model to ensure valid transitions between joint states. The FK module is adapted from (Holden et al., 2017) to be differentiable.
A.2 PROTOCOL TRAINING
Algorithm 1 Training Overview 1: Let G← set of communicative intents (goals) 2: Let A← population of agents 3: Randomly Initialize Θ,Φ #(policy and discriminator models, for all agents) 4: for i ∈ Iterations do 5: # Train Sender θ+ Receiver φ 6: B∼ G #(sample batch of intents, according to predefined distribution) 7: for g ∈ B do 8: s0← P0 #(agent reference pose) 9: for t ≤ T do 10: µ(s,θ)← π(st ‖ g) #(compute policy action - given state and target intent) 11: at ← µ(s,θ)+N(0, I)∗σ #(add noise - for exploration and imperfect actuation) 12: st+1← FK(st ,at) #(analytically compute next state - using given agent kinematics) 13: end for 14: τ ←{s1,a1,s2,a2, . . . ,sT } #(compose actor trajectory - sequence of joint states and actions) 15: ĝ← Dφ (τ) #(compute observer prediction of intent) 16: R =−L(g, ĝ) #(compute shared return) 17: Update model parameters Θ,Φ 18: end for 19: end for
A.3 EXPERIMENTAL SETUP
We design an Intent Recognition referential task for our experiments and consider a library of 10 communicative intents (concepts) given a priori (e.g. ”yes”, ”no”, ”turn left”, ”come here”). Each population is instantiated with a size of 10 agents. The agents are not situated in any physical or virtual environment. At each iteration of the game, a batch of 1024 intents is sampled, based upon the specified distribution (uniform or Zipf). Each sender agent generates a batch of 1024 intent-conditioned trajectories. Receiver agents are able to directly observe positions and orientations of sender joints. However, noise is added to each action, to represent imperfect observability. Noise emission is modeled through a unimodal Gaussian distribution N(0,σ), with fixed {σp = 2.0,σr = 0.4} values for joint positions and rotations, respectively. Here, position values are assumed to be measured in feet and rotation values in radians. Both policy and discriminator models are 3-layer feed-forward neural networks, with ReLU activations on all layers except the final. The discriminator network uses a SoftMax for the final layer, and the policy network uses no activation in the final layer, since it is allowed to output a broad range of continuous action values.
We experiment with a robot arm morphology in this work, motivated by potential application domains where non-verbal communication is most relevant, i.e. robotics and graphics. The agents in the population each have 4 articulated joints. No constraints were imposed on agent joints, so as to allow maximal range of motion. We did however employ action equivalences, whereby the modulus operator was applied to joint angles predicted by the policy network; this confined all joint angle values to the range of [0,2π]. We employed the same learning rate for both the policy and discriminator networks, 1e− 3. As a simplifying assumption, we consider a small time horizon (T = 5), but upsample the number of frames for visualization.
For the external observer evaluation, we randomly initialize a new discriminator model and split the original community, with 80% of the agents assigned to training set and 20% assigned to the test set. In each iteration, the training actors each generate a batch of 1024 intent-conditioned trajectories, based upon the sampled Zipf intent distribution. The external observer is also tested at each iteration with the trajectories generated by the test actors, who also each generate a batch of 1024 trajectories. The external observer experiments were run for 500 iterations on N = 2 and N = 5 concepts tasks and 1000 iterations on the N = 10 task, though learning stabilizes after approximately 100 iterations.
A.4 DISCRETE DOMAIN: TASK 1 – ADDITIONAL RESULTS
The first task in this simple discrete-channel domain considers 10 discrete actions each sender agent can take, each assigned its own unique energy value: Φ(a)= id(a). Examples of three independently
trained protocols for baseline (Zipf only) and experimental (Zipf + Energy Penalty) conditions of this task are visualized in Figures 7a and 8a, respectively. Visual illustrations of pairing off the senders with unseen receivers from the set in CP (for evaluation of ZS coordination) are shown in Figures 7b and 8b. These are juxtaposed with SP pairings and protocols trained (Figures 7a and 8a).
A.5 DISCRETE DOMAIN: TASK 2 (PROTOCOL LEARNING GIVEN ACTION DEGENERACIES)
The second task in the simple discrete communication domain also considers protocols with 5 intents and only five distinct energy values. However, there are now 17 discrete actions, which implies action redundancy. In particular, one action has a unique, lowest energy value; the 4 other energy values are each mapped to four different actions.
In Physics, a degeneracy denotes the number of distinct quantum states of a system that have a given energy. Analogously, this experiment investigates what happens in the degenerate case where several actions can be mapped to the same energy cost. Thus even in optimizing for minimal effort protocols, there are many ways to converge upon a globally optimal protocol, and consequently, ZS coordination becomes more challenging. This is because a sender-receiver pair can both maximize disriminability and minimize energy by assigning different actions (but with the same low energy exertion) to different intents.
This gets closer to challenges faced with continuous action protocols, where as the likelihoods of different intents grow arbitrarily close together, a sender may decide to employ visually distinct trajectories, but with comparable (low) energy exertions, to different intents. When this happens, it becomes exceedingly difficult to differentiate intents (for a new receiver agent) based upon the latent energy values, and ZS communication performance degrades.
Examples of learned protocols for baseline (Zipf only) and experimental (Zipf + Energy Penalty) conditions of this task are visualized and zoomed in on in Figures 9 and 10, respectively. Those same protocols are juxtaposed with visual illustrations of pairing off senders with unseen receivers for CP, in Figures 11 and 12.
For quantitative analysis on the value of inducing energy costs on top of a Zipfian distribution, for both conditions, we computed SP and CP performance between and across agent pairs, respectively. Like the simpler (first) task in this discrete domain, while SP performance at the end of protocol training is comparable for the two conditions, CP performance is notably different between them. In particular, CP success was 0.20±0.07 and 0.35±0.2 for the zipf (only) condition vs zipf + energy conditions, respectively. As an additional point of comparison, the most frequencly occuring class for N = 5 intents occurs with approximately 0.44 probability. So even in the experimental condition which performed better, ZS coordination capability does not exceed simply guess the maximally occurring class; this speaks to the inherent difficulty of learning ZS policies, even in seemingly simple domains. The empirical findings show evidence that on this harder communication task (with action degeneracies), ZS coordination is more difficult to achieve than as compared to the simpler task where all actions were assigned unique energy values. This was expected, but still useful to confirm. Lastly and importantly, these findings suggest that even when action degeneracies exist, coupling a nonuniform distribution with the energy objective is helpful for maximizing generality of communication skills learned. This finding is also consistent with what we observed on the simpler task in this discrete communication domain.


summaries: In this paper, the authors propose a technique for uncertainty estimation in regression with neural networks. The basic idea is to use an auxiliary "meta model" that (in the authors' best performing setting) has access to the base model and is trained jointly with it. The purpose of the meta model is to predict the error characteristics of the base model, which of course naturally leads to error bars. In order to account for the fact that the models errors on the train set are unlikely to be representative, the authors make use of a validation set on which the meta model is trained further.

summaries: In this work, the authors present a meta-modeling approach to provide predictions with uncertainty estimates in a sequential task. They develop a white box, black box, and joint modeling method that allows them to apply their method to a variety of scenarios. These methods differ based on the amount of information provided to a meta-learner which has the goal of predicting errors $\hat{z} \in \mathbb{R}^{D \times M} $. The authors also incorporate the ability to make asymmetric uncertainty bounds. They apply this method and many baeslines to two datasets: MITV and SPE9PR.

summaries: This paper describes a method to generate symmetric and asymmetric uncertainty estimates. The method is proposed to work for the non-stationarity processes found in real-world applications. The paper introduces a meta-modelling concept as an approach to achieve high-quality uncertainty quantification in deep neural networks for sequential regression tasks. The paper also introduces metrics for evaluating the proposed approach. A proposed meta-modelling approach is related to the work of Chen et al. (2019) which is mainly used for classification task in a non-sequential setting, however, the proposed method is mainly for the sequential setting.

abstractText: Generating high quality uncertainty estimates for sequential regression, particularly deep recurrent networks, remains a challenging and open problem. Existing approaches often make restrictive assumptions (such as stationarity) yet still perform poorly in practice, particularly in presence of real world non-stationary signals and drift. This paper describes a flexible method that can generate symmetric and asymmetric uncertainty estimates, makes no assumptions about stationarity, and outperforms competitive baselines on both drift and non drift scenarios. This work helps make sequential regression more effective and practical for use in real-world applications, and is a powerful new addition to the modeling toolbox for sequential uncertainty quantification in general.

authors:
M. Arnold
R.K.E. Bellamy
M. Hind
S. Houde
S. Mehta
A. Mojsilović
R. Nair
K.N. Ramamurthy
A. Olteanu
D. Piorkowski
D. Reimer
J. Richards
J. Tsay
K.R. Varshney

title: Factsheets: Increasing trust in ai services through supplier’s declarations of conformity

venue: IBM Journal of Research and Development,

authors:
E. Begoli
T. Bhattacharya
D Kusnezov

title: The need for uncertainty quantification in machineassisted medical decision making

venue: Nature Mach Intell,

authors:
Samy Bengio
Oriol Vinyals
Navdeep Jaitly
Noam Shazeer

title: Scheduled sampling for sequence prediction with recurrent neural networks

venue: In Proceedings of the 28th International Conference on Neural Information Processing Systems - Volume 1,

title: Generalized autoregressive conditional heteroskedasticity

venue: Journal of Econometrics,

authors:
Tongfei Chen
Jiřı́ Navrátil
Vijay Iyengar
Karthikeyan Shanmugam

title: Confidence scoring using whitebox meta-models with linear classifier probes

venue: In The 22nd International Conference on Artificial Intelligence and Statistics,

authors:
C. Chiu
T.N. Sainath
Y. Wu
R. Prabhavalkar
P. Nguyen
Z. Chen
A. Kannan
R.J. Weiss
K. Rao
E. Gonina
N. Jaitly
B. Li
J. Chorowski
M. Bacchiani

title: State-of-the-art speech recognition with sequence-to-sequence models

venue: IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP),

title: Modified randomization tests for nonparametric hypotheses

venue: Ann. Math. Statist.,

title: Autoregressive conditional heteroscedasticity with estimates of the variance of united kingdom

venue: inflation. Econometrica,

authors:
Chelsea Finn
Aravind Rajeswaran
Sham Kakade
Sergey Levine

title: Online meta-learning

venue: In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference on Machine Learning,

authors:
Yarin Gal
Zoubin Ghahramani

title: A theoretically grounded application of dropout in recurrent neural networks

venue: Advances in Neural Information Processing Systems

authors:
Chuan Guo
Geoff Pleiss
Yu Sun
Kilian Q. Weinberger

title: On calibration of modern neural networks

venue: In Proceedings of the 34th International Conference on Machine Learning - Volume 70,

authors:
Sepp Hochreiter
Jürgen Schmidhuber

title: Long short-term memory

venue: Neural Comput.,

authors:
Heinrich Jiang
Been Kim
Melody Guan
Maya Gupta

title: To trust or not to trust a classifier

authors:
Alex Kendall
Yarin Gal

title: What uncertainties do we need in bayesian deep learning for computer vision

title: Ninth SPE comparative solution project: A reexamination of black-oil simulation

venue: In SPE Reservoir Simulation Symposium. Society of Petroleum Engineers,

authors:
Diederik P. Kingma
Jimmy Ba

title: Adam: A method for stochastic optimization, 2014. cite arxiv:1412.6980Comment: Published as a conference paper at the 3rd International Conference for Learning Representations

venue: San Diego,

authors:
Balaji Lakshminarayanan
Alexander Pritzel
Charles Blundell

title: Simple and scalable predictive uncertainty estimation using deep ensembles

authors:
Aleksander Madry
Aleksandar Makelov
Ludwig Schmidt
Dimitris Tsipras
Adrian Vladu

title: Towards deep learning models resistant to adversarial

venue: attacks. ArXiv,

title: Time Series Techniques for Economists

authors:
D.A. Nix
A.S. Weigend

title: Estimating the mean and variance of the target probability distribution

venue: IEEE International Conference on Neural Networks (ICNN’94),

authors:
Min-hwan Oh
Peder A. Olsen
Karthikeyan Natesan Ramamurthy

title: Crowd counting with decomposed uncertainty

venue: In AAAI Conference on Artificial Intelligence,

authors:
A. Papoulis
H. Saunders

title: Probability, Random Variables and Stochastic Processes (2nd Edition)

venue: Journal of Vibration, Acoustics, Stress, and Reliability in Design, 111(1):123–125,

authors:
Steven J. Rennie
Etienne Marcheret
Youssef Mroueh
Jerret Ross
Vaibhava Goel

title: Self-critical sequence training for image captioning

title: Evolutionary principles in self-referential learning. on learning now to learn: The meta-meta-meta...-hook

venue: Diploma thesis, Technische Universitat Munchen, Germany,

authors:
Yanxia Shen
Xu Wang
Jie Chen

title: Wind power forecasting using multi-objective evolutionary algorithms for wavelet neural network-optimized prediction intervals

venue: Applied Sciences,

authors:
Jasper Snoek
Yaniv Ovadia
Emily Fertig
Balaji Lakshminarayanan
Sebastian Nowozin
D Sculley
Joshua Dillon
Jie Ren
Zachary Nado

title: Can you trust your model’s uncertainty? evaluating predictive uncertainty under dataset shift

authors:
Jose Sotelo
Soroush Mehri
Kundan Kumar
João Felipe Santos
Kyle Kastner
Aaron C. Courville
Yoshua Bengio

title: Char2wav: End-to-end speech synthesis

venue: In 5th International Conference on Learning Representations,

authors:
Nitish Srivastava
Geoffrey Hinton
Alex Krizhevsky
Ilya Sutskever
Ruslan Salakhutdinov

title: Dropout: A simple way to prevent neural networks from overfitting

venue: Journal of Machine Learning Research,

authors:
Ilya Sutskever
Oriol Vinyals
Quoc V. Le

title: Sequence to sequence learning with neural networks

venue: In Proceedings of the 27th International Conference on Neural Information Processing Systems Volume 2,

authors:
Zoltán Tüske
Kartik Audhkhasi
George Saon

title: Advancing sequence-to-sequence based speech recognition

venue: Proc. Interspeech 2019,

heading: 1 INTRODUCTION

text:
The ability to quantify the uncertainty of a model is one of the fundamental requirements in trusted, safe, and actionable AI (Arnold et al., 2019; Jiang et al., 2018; Begoli et al., 2019).
This paper focuses on uncertainty quantification in regression tasks, particularly in the context of deep neural networks (DNN). We define a sequential task as one involving an ordered series of input elements, represented by features, and an ordered series of outputs. In sequential regression tasks (SRT), the output elements are (possibly multivariate) real-valued variables. SRT occur in numerous applications, among others, in weather modeling, environmental modeling, energy optimization, and medical applications. When the cost of making an incorrect prediction is particularly high, such as in human safety, models without a reliable uncertainty estimation are perceived high risk and may not be adopted.
Uncertainty prediction in DNNs has been subject to active research, in particular, spurred by what has become known as the “Overconfidence Problem” of DNNs Guo et al. (2017), and by their susceptibility to adversarial attacks Madry et al. (2017). However, the bulk of work is concerned with non-sequential, classification tasks (see Section 2) leaving a noticeable gap for SRT.
In this paper we introduce a meta-modeling concept as an approach to achieving high-quality uncertainty quantification in DNNs for SRT. We demonstrate that it not only outperforms competitive baselines but also provides consistent results across a variety of drift scenarios. We believe the approach represents a new powerful addition to the modeling toolbox in general.
The novel contributions of this paper are summarized as follows: (1) Application of the meta-modeling concept to SRT, (2) Developing a joint base-meta model along with a comparison to white- and black-box alternatives, (3) Generating asymmetric uncertainty bounds in DNNs, and (4) Proposing a new evaluation methodology for SRT.

heading: 2 RELATED WORK

text:
Classical statistics on time series offers an abundance of work dealing with uncertainty quantification (Papoulis & Saunders, 1989). Most notably in econometrics, a variety of heteroskedastic variance models lead to highly successful application in financial market volatility analyses (Engle, 1982; Bollerslev, 1986; Mills, 1991). An Autoregressive Conditional Heteroskedastic, or ARCH, model
(Engle, 1982), and its generalized version, GARCH, (Bollerslev, 1986) are two such methods, the latter of which serves as one of our baselines.
An illuminating study (Kendall & Gal, 2017) describes an integration of two sources of uncertainty, namely the epistemic (due to model) and the aleatoric (due to data). The authors propose a variational approximation of Bayesian Neural Networks and an implicit Gaussian model to quantify both types of variability in a non-sequential classification and regression task. Based on Nix & Weigend (1994), Lakshminarayanan et al. (2017) also uses an implicit Gaussian model to improve the predictive performance of a base model, again in a non-sequential setting. Similar to Kendall & Gal (2017), the study does not focus on comparing the quality of the uncertainty to one generated by other methods. We use the implicit variance model of Kendall & Gal (2017); Oh et al. (2020); Lakshminarayanan et al. (2017), as well as the method of variational dropout of Gal & Ghahramani (2016); Kendall & Gal (2017) as baselines in our work. A meta-modeling approach was taken in Chen et al. (2019) aiming at the task of instance filtering using white-box models. The work relates to ours through the meta-modeling concept but concentrates on classification in a non-sequential setting. Besides its application in filtering, meta-modeling has been widely applied in the task of learning to learn and lifelong learning (Schmidhuber, 1987; Finn et al., 2019). However, it should be pointed out that the two applications of meta-modeling are not comparable due to their different objectives. Uncertainty in data drift conditions was assessed in a recent study (Snoek et al., 2019). The authors employ calibration-based metrics to examine various methods for uncertainty in classification tasks (image and text data), and conclude, among others that most methods’ quality degrades with drift. Acknowledging drift as an important experimental aspect, our study takes it into account by testing in matched and drifted scenarios. Finally, Shen et al. (2018) described a multi-objective training of a DNN in wind power prediction, minimizing two types of cost related to coverage and bandwidth. We expand on these metrics in Section 3.3.

heading: 3 METHOD

heading: 3.1 META MODELING APPROACH

text:
The basic concept of Meta Modeling (MM), depicted in Figure 1, involves a combination of two models comprising a base model, performing the main task (e.g., regression), and a meta model, learning to predict the base model’s error behavior. Depending on the amount of information shared between these two, we distinguish several settings, namely (1) base model is a black-box (BB), (2) base is a white-box (WB, base parameters are accessible), and (3) base and meta components are trained jointly (JM). The advantages of WB and JM are obvious: rich information is available for the meta model to capture salient patterns for it to generate accurate predictions. On the other hand, the BB setting often occurs in practice and is a given.
We now formalize the MM concept as it applies to sequential regression. Let ŷ = Fφ(x) be the base model function parametrized by φ, where x = x1, ..., xN and ŷ = ŷ1, ..., ŷM represent sequences of N input feature vectors and M Ddimensional output vectors, with ŷ ∈ RD×M . Let ẑ = Gγ(ŷ,x, φ) denote the meta model, parameterized by γ, taking as input the pre-
dictions, the original features, and the parameters of the base to produce a sequence of error predictions, ẑ ∈ RD×M . The parameters φ are obtained by solving an optimization problem, argminφ E[lb(ŷ,y)], using a smooth loss function lb, e.g., the Frobenius norm lb = ‖ŷ − y‖ 2 F . Similarly, the parameters γ are determined via argminγ E[lm(ẑ, z)] = argminγ E[lm(ẑ, lz(ŷ,y))] involving a loss lz quantifying the target error (residual) from the base model, and lm quantifying the prediction error of the meta model. In general, lb, lz , and lm, may differ. The expectations are estimated using an available dataset. We used the L2F norm for lb and lm, and L1 for lz , as described in Section 4. Given differentiable loss functions and the DNN setting, the base and the meta model can be integrated in a single network (JM). In this case the parameters are estimated jointly via
φ∗, γ∗ = argmin φ,γ E[βlb(ŷ,y) + (1− β)lm(ẑ, lz(ŷ,y))] (1)
whereby dedicated output nodes of the network generate ŷt and ẑt, and β is a hyper-parameter trading off the base with the meta loss. Thus, one part of the network tackles the base task, minimizing the base residual, while another models the residual as the eventual measure of uncertainty. As done in (Kendall & Gal, 2017), one can argue that the base objective minimizes the epistemic (parametric) uncertainty, while the meta objective captures the aleatoric uncertainty present in the data. Due to their interaction, the base loss is influenced by the estimated uncertainty encouraging it to focus on feature-space regions with lower aleatoric uncertainty. Moreover, we conjecture, the DNN base model is encouraged to encode the input in ways suitable for uncertainty quantification.
Figure 2 shows an overview of a sequential DNN architecture applied throughout our study. It includes a base encoder-decoder pair and a meta decoder connected to them. Each of these contains a recurrent memory cell - the LSTM (Hochreiter & Schmidhuber, 1997). The role of the encoder is to process the sequential input, x, compress its information in a context vector and pass it to the base decoder. The recurrent decoder produces the regression output ŷ in M time steps feeding its predictions as input in the next time steps. Evolving in time, both base LSTMs update their internal states bt and ht, whereby the last state, bN , serves as the context vector for the decoder. This architecture has gained wide popularity in applications such as speech-to-text (Chiu et al., 2018; Tüske et al., 2019), text-to-speech (Sotelo et al., 2017), machine translation (Sutskever et al., 2014), and image captioning (Rennie et al., 2016). Following the MM concept, we attach an additional decoder (the meta decoder) via connections to the encoder and decoder outputs. The context vector, bN , is transformed by a fully connected layer (FCN in Figure 2), and both the ŷt output as well as the internal state, ht, are fed into the meta component. As mentioned above, the meta decoder generates uncertainty estimates, ẑt.
Given the architecture depicted in Figure 2, we summarize the three settings as follows:(1) Joint Model (JM): parameters are trained according to Eq. (1) with certain values of β. (2) White-Box model (WB): base parameters φ are trained first, followed by parameters γ, also accessing φ. (3) Black-Box model (BB): same as (2) without access to φ.
Figure 2: Encoder-Decoder architecture integrating a base and a meta model
Figure 3: Bandwidth, excess, and deficit costs.
Generating Symmetric and Asymmetric Bounds The choice of loss function, lz , gives rise to two scenarios. If lz is an even function, e.g., lz(ŷ,y) = ‖ŷ − y‖1, the meta-model targets z capture base error equally in both directions: above and below the target. Hence, the uncertainty ẑ predicted at test time will represent a interval symmetric around ŷ. If, on the other hand, lz takes the sign in z into account, it is possible to dedicate separate network nodes γl, γu ∈ γ to capturing lower and upper band estimates, ẑl and ẑu, respectively, thus accomplishing asymmetric prediction. Let δ = ŷ − y. For the asymmetric scenario the meta objective is modified as follows:
γ∗ = argmin γ E[lm(zl,max{δ, 0}) + lm(zu,max{−δ, 0})] (2)

heading: 3.2 BASELINES

text:
Implicit Heteroskedastic Variance Lakshminarayanan et al. (2017); Kendall & Gal (2017); Oh et al. (2020) applied a Gaussian modelN (µ, σ2) to the output of a neural network predictor, where µ
represents the prediction and σ2 its uncertainty due to observational (aleatoric) noise. The model is trained to minimize the negative log-likelihood (NLL), with the variance being an implicit uncertainty parameter (in that it is trained indirectly) which is allowed to vary across the feature space (heteroskedasticity). We apply the Gaussian in the sequential setting by planting it onto the base decoder’s output (replacing the meta decoder) and train the network using the NLL objective: φ∗ = argminφ E [∑M t=1 ∑D d=1 (ŷt,d−yt,d)2 σ2t,d + log σ2t,d ] with D output nodes modeling the regression variable, ŷt, and separate D output nodes modeling the log σ2t , at time t.
Variational Dropout Gal & Ghahramani (2016) established a connection between dropout (Srivastava et al., 2014), i.e., the process of randomly omitting network connections, and an approximate Bayesian inference. We apply the variational dropout method to the base encoder and decoder. By performing multiple runs per test sequence, each with a different random dropout pattern, the base predictions are calculated as the mean and the base uncertainty as the variance over such runs. This Bayesian method, along with the variational approximation, captures the parametric (epistemic) uncertainty of the model, hence it fundamentally differs from the Gaussian model as well as our proposed approach.
GARCH Variance Introduced in (Bollerslev, 1986; Engle, 1982), the Generalized Autoregressive Conditional Heteroskedastic (GARCH) variance model belongs among the most popular statistical methods. A GARCH(p,q) assumes the series to follow an autoregressive moving average model and estimates the variance at time t as a linear combination of past q residual terms, 2, and p previous variances, σ2: σ2t = α0 + ∑q i=1 αi 2 t−i + ∑p i=0 βiσ 2 t−i.
The α0 term represents a constant component of the variance. The parameters, α, β are estimated via maximum-likelihood on a training set. The GARCH process relates to the concept in Figure 1 in that it acts as the meta-model predicting the squared residual. We use the GARCH as a baseline only on one of the datasets for reasons discussed in Section 4.
Constant-Band Baseline A consistent comparison of uncertainty methods is difficult due to the fact that each generates an uncertainty around different base predictions. Therefore, as a reference we also generate a constant symmetric band around each base predictor. Such a bound represents a homoskedastic process – a sensible choice in many well-behaved sequential regression problems, corresponding to a GARCH(0,0) model. We will use this reference point to compute a relative gain of each method as explained in Section 3.3.

heading: 3.3 EVALUATION METHODOLOGY

text:
Core Metrics Unlike with classification tasks, where standard calibration-based metrics apply (Snoek et al., 2019), we need to consider two aspects arising in regression, roughly speaking: (1) what is the extent of observations falling outside the uncertainty bounds (Type 1 cost), and (2) how excessive are the bounds (Type 2 cost). An optimal bound captures all of the observation while being least excessive in terms of its bandwidth. Shen et al. (2018), among others, defined two measures reflecting these aspects (miss rate and bandwidth) which we adopt below (Eqs. (3) and (4)) while adding two refinements (Eqs. (5) and (6)). Let ŷl = ŷ − ẑl and ŷu = ŷ + ẑu denote the predicted lower and upper bound, respectively. Recall that ŷ ∈ RD×M . We define the following metrics:
Missrate(ŷl, ŷu,y) = 1− 1 MD ∑ d,t:ydt∈[ŷldt,ŷ u dt] 1 (3)
Bandwidth(ŷl, ŷu,y) = 1
2MD D∑ d=1 M∑ t=1 ŷudt − ŷldt (4)
Excess(ŷl, ŷu,y) = 1
MD ∑ d,t:ydt∈[ŷldt,ŷ u dt] min { ydt − ŷldt, ŷudt − ydt } (5)
Deficit(ŷl, ŷu,y) = 1
MD ∑ d,t:ydt /∈[ŷldt,ŷ u dt] min { |ydt − ŷldt|, |ydt − ŷudt| } (6)
Figure 3 illustrates these metrics. The relative proportion of observations lying outside the bounds (miss rate) ignores the extent of the bound’s short fall. The Deficit, Eq. (6), captures this. The type 2 cost is captured by the Bandwidth, Eq. (4). However, its range is indirectly compounded by the underlying variation in ŷ and y. Therefore we propose the Excess measure, Eq. (5), which also reflects the Type 2 cost, but just the portion above the minimum bandwidth necessary to include the observation.
Calibration In general, DNNs offer few guarantees about the behavior of their output. DNNs tend to produce miscalibrated classification probabilities (Guo et al., 2017). In order to evaluate the uncertainty across models, it is necessary to establish a common operating point (OP). We achieve this via a scaling calibration. For symmetric bounds, we assume that y = ŷ + ẑ where ∈ RD×M is a random i.i.d. matrix and ẑ is the predicted non-negative uncertainty band. Let Zdt =
ydt−ŷdt ẑdt . Using a held-out dataset, we can obtain an empirical distribution in each output
dimension: cdf({Zdt}1≤t≤M ), d = 1, ..., D. It is then possible to find the value ε (p) d for a desired quantile p, e.g., ε(0.95)d , and construct the prediction bound at test time: [ ŷdt − ẑdtε(p)d , ŷdt + ẑdtε (p) d ] . Assuming Zd is stationary, in expectation, this bound will contain the desired proportion p of the observations, i.e., E[Missrate(ŷl, ŷu,y)] = 1− p. This scaling is applied in our evaluation to compare the excess, bandwidth and deficit at fixed miss rates as well as setting a minimum cost OP (see Section 3.3). An Algorithm to find a scale factor for a desired value of any of the four metrics in O(M2) operations is given in the Appendix.
Metrics Used in Reporting The following OP-based measures are used in reporting: (1) Excess, Deficit, Bandwidth at a fixed Missrate, averaged over Missrate = {0.1, 0.05, 0.01}, and (2) Minimum Excess-Deficit cost, where cost = 12 (Excess + Deficit) with the minimum found over all calibrations (OPs).
For each system and measure, ms, a symmetric constant-band baseline, mfixed, is also generated and a relative gain with respect to this reference calculated: gains = 100× mfixed−msmfixed %. Finally,
the error rate of the base predictor is calculated as Ebase(ŷ,y) = 1D ∑D d=1 ‖ŷd−yd‖1 ‖yd‖1
, where ŷd, yd are d-th row vectors of ŷ,y.

heading: 4 EXPERIMENTS

text:
Datasets Two sequential regression datasets, namely the Metro Interstate Traffic Volume (MITV) dataset1 and the SPE9 Reservoir Production Rates (SPE9PR) dataset2, were experimented with. Both originate from real-world applications, involve sequential input/output variables, and provide for scenarios with varying degrees of difficulty.
The MITV dataset is a collection of hourly weather features with the target of the regression being the hourly traffic volume, recorded by the Minnesota DoT continuously between 2012 and 2018. The SPE9PR, on the other hand, is a large collection of mathematical simulations of a reservoir field with varying input and output sequences with each simulation comprising a sequence with 100 time steps. The regression targets in this case are multivariate and correspond to field production rates.
1https://archive.ics.uci.edu/ml/machine-learning-databases/00492/ 2https://developer.ibm.com/technologies/artificial-intelligence/data/oil-reservoir-simulations
The SPE9PR also contains a data partition collected under distributional drift. Full detail on both datasets including their preprocessing can be found in the Appendix.
Training Procedure Each dataset was partitioned into TRAIN, DEV, DEV2, and TEST sets (with SPE9PR also providing a TEST-drift set), as listed Table 1. While the TRAIN/DEV partitions served basic training, the DEV2 was used in determining hyperparameters and operating points (calibration). The TEST sets were used to produce the reported metrics. While a single sample in the SPE9PR represents a complete sequence of 100 steps, the MITV data come as a single contiguous sequence. The partitioning of the MITV set is strictly ordered by time, whereby the DEV sequence follows TRAIN, DEV2 follows DEV, and TEST follows DEV2. After partitioning, each MITV sequence was processed by a sliding window of length 36 hours (in 1-hour steps). This resulted in a series of (n − 35) × 36 subsequences (n denotes the partition size) to feed the encoder-decoder model. When testing on the MITV, DNN predictions from such sliding windows were recombined into a contiguous prediction sequence again.
The base encoder-decoder network (see Figure 2) is trained using the Adam optimizer (Kingma & Ba, 2014) with a varying initial learning rate, lr, in two stages: (1) Training of all parameters using TRAIN while providing the ground truth as the decoder input at each time step. (2) Building on the previous, the training continues, however, decoder predictions from step t − 1 are fed as decoder inputs at step t—a mode referred to as emulation by Bengio et al. (2015). All hyperparameter values are listed in the Appendix.
JOINT MODEL, SYMMETRIC (JMS), AND ASYMMETRIC (JMA): The common training steps are performed using the objective in Eq. (1), with β = 1.0, first. Then, the joint training continues with β = 0.5 as long as the objective improves on DEV. In a final step, the model switches to using DEV as training with β = 0.0 until no improvement on TRAIN is seen. A similar procedure is followed for the JMA, except using Eq. (2). WHITE-BOX MODEL, SYMMETRIC (WBMS): The basic training is performed with β = 1.0. Next, only meta model parameters are estimated using the DEV/TRAIN sets with β = 0.0. BLACK-BOX MODEL, SYMMETRIC (BBMS): Base training is performed. The base model processes the DEV set to generate residual z. A separate encoder-decoder model is then trained using (x, z). JOINT MODEL WITH VARIANCE (JMV): The two common steps are performed using the NLL objective (see Section 3.2) with the variance-related parameters first fixed, and, in a subsequent step, allowing the variance parameters to be adjusted, until convergence. This is to aid stability in training (Nix & Weigend, 1994). DROPOUT MODEL SYMMETRIC (DOMS): Dropout with a rate of 0.25 (inputs, outputs) and 0.1 (internal states), determined as best performing on the DEV2 set, were applied in both the base encoder and the decoder. The two common steps were performed. At test time, the model was run 10 times per each test sequence to obtain the mean prediction and standard deviation. GARCH: The GARCH model from Section 3.2 is used with p = q = 5. The lag value was determined using an autocorrelation chart showing attenuation at lags > 5. We only apply this baseline to the MITV dataset as it provides a contiguous time series. The model parameters were trained using the DEV2 partition.
Throughout the experiments, the size of each LSTM cell was kept fixed at 32 for the base encoder/decoder, and at 16 for the meta decoder. The base sizing has been largely driven by our preliminary study showing it suffices in providing accurate base predictions.
Testing Procedure As mentioned above the SPE9PR dataset has two TEST partitions: one for a matched and one for a drifted condition. While the MITV dataset does not provide an explicit source of drift, we induce drift by creating a discrepancy in the modeling procedure between training and test: In the non-drift condition, the DNN’s decoder is given access to the past 12 hours worth of traffic observations to make a forecast for the next 24 hours. This is achieved by spanning a 36-hour window and feeding the decoder inputs the first 12 hours of ground truth, during training. Now to create the drift scenario we test the MITV model without providing those first 12 hours of observations and the model uses their own predictions for that period instead. This emulates a “model drift” condition in that the model, trained to rely on actual observations, is getting its own noisy predictions.
Results with Symmetric Bounds Table 2 compares the proposed symmetric-bounds systems (JMS, WBMS, BBMS) with the baselines (JMV, DOMS, GARCH). The relative error of the base predictor is given in the Ebase column. The uncertainty quality is reported in Table 2 is the average gain in excess-deficit metrics, as defined in Section 3.3. Columns labeled as G∗ contain measurements made at an operating point (OP) determined on the test set itself, while those labeled as Gx use an OP from a held-out (DEV2) set. While Gx reflects generalization of the calibration, G∗ values are interesting as they reveal the potential of each method. Based on a paired permutation test (Dwass, 1957) all but entries marked with † are mutually significant at p < 0.01. From Table 2 we make the following observations: (1) the JMS model dominates all other models across all conditions. The fact that it outperforms the WBMS indicates there is a benefit to the joint training setup, as conjectured earlier. (2) The WBMS dramatically outperforms the BBMS model, which remains only as good as a constant band for MITV data, indicating it is hard to reliably predict residuals from only the input features. (3) The most competitive baseline is the JMV model. As discussed in Section 3.2, the JMV shares some similarity with the meta-modeling approach. (4) The JMS and WBMS models perform particularly well in the strong drift scenario (SPE9PR), suggesting that white-box features play an essential role in achieving generalization. Finally, (5) the DOMS model works well on MITV data but provides no benefit in the SPE9PR, which could be due the aleatoric uncertainty playing a dominant role in this dataset. In almost all cases, however, the averaging of base predictions in DOMS results in lowest error rates of the base predictor.
Representative samples of JMS and JMV uncertainty bounds are shown in Figure 4 (MITV) and Figure 5 (SPE9PR). They illustrate a clear trend in the results, namely that the JMS (also seen with WBMS) model are better able to cover the actual observation, particularly when the base prediction tends to make large errors. Additional plots can be found in the Appendix, and also a notebook to visualize all test samples is provided as part of the Supplementary Material.
Results with Asymmetric Bounds Generating asymmetric bounds is a new intriguing aspect of DNN-based meta-models. Using the JMA model, we first recorded the accuracy with which the
asymmetric output agrees in sign (orientation) with the observed base discrepancy. Averaged over each of the two datasets, this accuracy is at 83.3%, and 91.1%. The promise of asymmetric bounds lies in its potential to reduce the bandwidth cost. Since the Excess and Deficit metrics ignore the absolute bandwidth, we also evaluate the JMA model using the Bandwidth metric (Eq. (4)), averaged over the same OPs. The results are shown in Table 3 comparing the JMA model to the best symmetric model, JMS. The JMS model outperforms JMA in all scenarios on Excess-Deficit, however, compared on the bandwidth metric, the JMA dominates benefiting from its orientation capability. Upon visual inspection the output of the JMA is appreciably better in bandwidth: Figure 6 shows samples on both datasets. In most instances the bounds behave as expected, expending the bulk of bandwidth in the correct direction. An interesting question arises whether it is possible to utilize the asymmetric output as a correction on the base predictor. Our preliminary investigation shows that a naive combination leads to degradation in the base error, however, this question remains of interest for future work.

heading: 5 CONCLUSIONS

text: In this work we demonstrated that meta-modeling (MM) provides a powerful new framework for uncertainty prediction. Through a systematic evaluation of the proposed MM variants we report considerable relative gains over a constant reference baseline and show that they not only outperform all competitive baselines but also show stability across drift scenarios. A jointly trained model integrating the base with a meta component fares best, followed by a white-box setup, indicating that trainable white-box features play an essential role in the task. Besides symmetric uncertainty, we also investigated generating asymmetric bounds using dedicated network nodes and showed their benefit in reducing the uncertainty bandwidth. We believe these results open an exciting new research avenue for uncertainty quantification in sequential regression.

heading: A ALGORITHM TO FIND A SCALING FACTOR

text: Section 3.3 discusses the scaling calibration in the context of the four metrics: Missrate, Bandwidth, Excess, and Deficit. Algorithm 1 finds a scale factor for a desired value of any of these four metrics in O(M2) operations.

heading: B ADDITIONAL DATASET AND IMPLEMENTATION DETAILS

heading: B.1 DATASETS

heading: B.1.1 METRO INTERSTATE TRAFFIC VOLUME (MITV)

text:
The dataset is a collection of hourly westbound-traffic volume measurements on Interstate 94 reported by the Minnesota DoT ATR station 301 between the years 2012 and 2018. These measurements are aligned with hourly weather features3 as well as holiday information, also part of the dataset. The target of regression is the hourly traffic volume. This dataset was released in May, 2019.
The MITV input features were preprocessed to convert all categorical features to trainable vector embeddings, as outlined in Figure 2. All real-valued features as well as the regression output were standardized before modeling (with the test predictions restored to their original range before calculating final metrics). Overall dataset statistics are listed in Table 1 and further processing steps are given in Section 4.

heading: B.1.2 MITV PREPROCESSING

text: As described in Section B.1.1 and Table 1, the MITV dataset comes with 8 input features, among which 3 are categorical. Here we list the relevant parsing and encoding steps used in our setup. The raw time stamp information was parsed to extract additional features such as day of the week, day of the month, year-day fraction, etc. Table 4 shows the corresponding list. Standardization was performed on the input as well as output, as per Table 4, whereby the model predictions were transformed to their original range before calculating final metrics.

heading: B.1.3 SPE9 RESERVOIR PRODUCTION RATES (SPE9PR)

text:
This dataset originates from an application of oil reservoir modeling. A reservoir model (RM) is a space-discretized approximation of a geological region subject to modeling. Given a sequence of drilling actions (input), a physics-based PDE-solver (simulator) is applied to the RM to generate
3provided by OpenWeatherMap

heading: INPUT

heading: OUTPUT

text: sequences of future production rates (oil, gas, water production), typically over long horizons Killough (1995). The objective is to train a DNN and accurately predict outputs on unseen input sequences. We used the publicly available SPE94 RM, considered a reference for benchmarking reservoir simulation in the industry, and an open-source simulator5 to produce 28,000 simulations, each with 100 randomized actions (varying type, location, and control parameters of a well) inducing production rate sequences over a span of 25 years, in 90-day increments, i.e., 100 time steps. Furthermore, the RM was partitioned into two regions, A and B. While most of the actions are located in the region A, we also generated 1000 sequences with actions located in the region B thus creating a large degree of mismatch between training and test. The test condition in region B will be referred to as “drift” scenario.

heading: B.1.4 SPE9PR PREPROCESSING

text:
The Table 5 lists details on the SPE9PR features (also refer to Section B.1.3 and Table 1). The SPE9PR dataset contains input sequences of actions and output sequences of production rates. An action (feature type of well), at a particular time, represents a decision whether to drill, and if so, what type of well to drill (an injector or a producer well), or not to drill (encoded by ”0”), hence the cardinality is 3. In case of a drill decision, further specifications apply, namely the x- and y-location on the surface of the reservoir, local geological features at the site, and well control parameters. There are 15 vertical cells in the SPE9 each coming with 3 geological features (rel. permeability, rel. porosity, rock type), thus the local geology is a 45-dimensional feature vector at a particular (x, y) location. Finally, every well drilled so far may be controlled by a parameter called ”Bottom-Hole Pressure” (BHP). Since we provision up to 100 wells of each of the two types, a 200-dimensional vector arises containing BHP values for these wells at any given time. Standardization was performed on the input as well as output as specified in Table 5 whereby the model predictions were transformed to their original range before calculating and reporting final metrics.
4https://github.com/OPM/opm-data/blob/master/spe9/SPE9.DATA 5https://opm-project.org/

heading: B.2 TRAINING SETUP

heading: B.2.1 HYPERPARAMETERS

text:
Hyperparameters have been determined in two ways: (1) learning rate, regularization, batch, and LSTM size were adopted from an unrelated experimental study performed on a modified reservoir SPE9 (Anonymized), (2) We used DEV2 to determine the dropout rates in the DOMS model. The value β = 0.5 was chosen ad-hoc (as a midpoint between pure base and pure meta loss) without further optimization.
B.3 IMPLEMENTATION NOTES
All DNNs were implemented in Tensorflow 1.11. Training was done on a Tesla K80 GPU, with total training time ranging between 3 (MITV) and 24 (SPE9PR) hours. The GARCH Python implementation provided in the arch library was used.

heading: C ADDITIONAL RESULTS

text:
C.1 INDIVIDUAL METRICS
For a more detailed view of the averages in Table 2, we show a split by the individual metrics in Table 9, and, for the SPE9PR which has a total of four output variables, a split by the individual variables in Table 10.

heading: C.2 ADDITIONAL VISUALIZATIONS

text:
In addition to the sample visualizations shown in Section 2 for the JMS, JMV, and JMA systems, here we show same sections of the data and visualize output of all systems. Figures 7 and 8 show the first simulation in the test set of the SPE9PR dataset for the drift and non-drift condition and all its output components, respectively. Figures 9 and 10 show the output on the MITV drift and non-drift condition, respectively. For each model, a miss rate value of 0.1 across the entire test set was used in the visualizations.
C.2.1 INTERACTIVE NOTEBOOK
We also provide an interactive notebook that allows for inspecting all system output on an arbitrary portion of the test data in both the non-drift and drift condition. Please refer to the README file within the zip-file uploaded as the Supplementary Material part of our submission.

heading: C.2.2 JMV VARIANCE TUNING ON DEV DATA

text:
Section 4 lists individual training steps for each system. It is noted that the meta-modeling arrangements have used the DEV partition for tuning in a final step. The motivation for using a partition not included in training the base model is the avoidance of meta-training on biased targets, i.e., targets generated by the base model on its own training data. In this context, a question arises whether a sim-
ilar tuning step could help the JMV model. We followed the training steps described in Section 4 and then updated the network nodes tied to the variance parameter while keeping the rest of the network fixed. The Table 11 shows the results on the MITV dataset. It seems the benefit of the tuning step does not materialize. In all but the cross-validated drift case the gain decreases (albeit insignificantly) when applying the DEV-only tuning. We conjecture that the benefit of the tuning step exists with the meta-model because of the direct supervision of the meta-model’s prediction. In contrast, the variance in the JMV setting is learned implicitly and may not suffer from the ”biased-target” problem mentioned above.