## Towards Learning How to Properly Play UNO with the iCub Robot

Pablo Barros 1 , Stefan Wermter 1 , and Alessandra Sciutti 2

1

Knowledge Technology, University of Hamburg, Hamburg, Germany Email: { barros, wermter @informatik.uni-hamburg.de }

Cognitive Architecture of Collaborative Technologies Unit, Istituto Italiano di Tecnologia, Genoa, Italy Email: alessandra.sciutti@iit.it

2

Abstract -While interacting with another person, our reactions and behavior are much affected by the emotional changes within the temporal context of the interaction. Our intrinsic affective appraisal comprising perception, self-assessment, and the affective memories with similar social experiences will drive specific, and in most cases addressed as proper, reactions within the interaction. This paper proposes the roadmap for the development of multimodal research which aims to empower a robot with the capability to provide proper social responses in a Human-Robot Interaction (HRI) scenario.

Index Terms -Human-Robot Interaction, Emotion Recognition, Affective Response

## I. INTRODUCTION

Our capabilities of both perceiving and reacting to the affective behavior of other persons are fine-tuned based on the observed social response of our interaction peers. We usually perceive how others are behaving towards us by reading their affective behavior through the processing of audio/visual cues [13]. We are then able to provide an affective response, either by modulating our voice, using facial expressions, producing body movements, or realizing a specific action within the interaction context. The response is usually modulated by a contagion effect [11], where we are impacted by and replicate the perceived affective state of others, or by reacting with a proper social response [10], which is accepted and expected by the interaction peers. Deciding the proper response is a learning process modulated by the interpretation of the impact that our behavior causes within the interaction context [17]. The learning association between perception and action configures a long-term adaptation mechanism which updates how we perceive and respond to others over a lifelong span. Adapting such a mechanism into a robotic platform would allow it to provide socially acceptable responses when in interactive social scenarios, and would benefit greatly the acceptability and human-likeliness of companion and assistive robots [9].

networks [6]. Such models can learn, implicitly, how to represent high-level stimuli, such as facial expressions and prosody in speech. Such capability allows these models to represent multimodal emotion expressions efficiently and robustly, and they currently post as the state-of-the-art solutions in different emotion recognition tasks. When deployed to real-world scenarios, in particular, involving social robots, however, these solutions tend to fail, mostly due to their inability to adapt quickly to how specific persons express emotions [19]. Deep learning models have millions of different parameters which need to be updated to depict different affective characteristics. Given the subjectivity on perceiving and labeling emotion expressions, to update these models to learn the specific characteristics of how a person expresses emotions without a large number of training samples of that specific person posts as a machine learning challenge.

Adapting this perception-action mechanism into a social robot demands the development and integration of two mechanisms: an adaptive emotion appraisal, which is updated over time towards novel persons, and a behavior generation which learns based on the perception of the social impact that a certain action caused within the interaction.

The state-of-the-art in emotion recognition for social robots was much benefited with the recent advents of deep neural

Generating behavior based on affective perception has been deeply studied by the social robots area [12]. Most of the proposed models, however, employ an emotion contagion strategy which usually reproduces the perceived facial expressions or body movement [22], or by using a simple decision tree on how to behave when a certain emotion was perceived [21]. Such solutions are ineffective when dealing with complex interaction scenarios, such as interactions which are based on empathetic responses, very common to conversational robots and robots are driven by social responses [14]. As soon as the robot has to provide a response which was not scripted by predefined decision mechanisms, it will usually fail to maintain an human-level of engagement with the users [20], which was observed to reduce its applicability when interacting with children or elderly citizens in therapy or health-care scenarios [5]. One of the most common machine learning solutions to generate robot behavior is to provide a learning mechanism based on reinforcement learning. Usually, in such learning scheme, a robot reads the feedback from the environment and use it in a reward function to modulate its learning. The advantage of using reinforcement learning is that the robot can learn by itself, with little or no human supervision, by measuring the effect that its actions caused in the environment. When deployed in Human-Robot Interaction (HRI) scenarios, however, reinforcement learning strategies are usually not effective as they demand several thousands of examples to learn associations between perception and actions. As in most

Fig. 1. The iCub humanoid robot. We will use it in our proposed scenario due to its social capabilities, both for the perception of audio/visual cues and the generation of facial expressions and body movement.

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.00744v1-figure-fe6622a3-33e6-4518-9bf7-cdc3e97dde50.jpg)

social interaction scenarios the robot interacts with a human, thousands of interactions would be necessary before the robot learns meaningful responses [8], reducing its applicability in real-world scenarios.

This position paper focuses on a multi-subject interaction scenario where different humans play a competitive card game, known as UNO, with a robot. The robot will be an active part of the interaction and will play against humans. As such, the robot will perceive the individual and group affective behavior and the game status and derive an action which is composed of behavioral response (facial expression and body movement), and a game-related decision (e.g. play a card or collect a card).

We also describe in this paper the first steps towards a hybrid neural architecture to be embedded into an iCub robot [15], illustrated in Figure 1. We drive the development of the proposed framework based on two directions: an adaptive perception neural network which uses an unsupervised affective memory mechanism to overcome the problems of online learning in deep neural networks to adapt towards how specific persons express emotions, and to model an intrinsic affective state; and an interactive reinforcement learning strategy based on learning the social and contextual impact of a certain action by measuring the affective responses of the players, avoiding the necessity of active human interaction in the learning loop.

This paper details the road-map on how the individual models that compose the framework will be developed and integrated. We also describe, and justify, in details the evaluation scenario and protocol to obtain a set of objective evaluation measures to evaluate the impact of our proposed framework.

## II. PLAYING UNO WITH A ROBOT

The iCub robot is one of the most advanced platforms for Human-Robot Interaction (HRI). It was evaluated in different interaction scenarios and it provides a high-level of engagement, acceptability, and reliability when interacting with different persons. These are important characteristics to improve the perception of the robot as an active social member in our scenario, which we assume it will be an important characteristic to elicit natural affective behavior.

In our proposed interaction scenario, the iCub is playing the card game named UNO with three other persons. In this

Fig. 2. Our competitive game scenario involving three persons and the iCub robot. The robot will read audio/visual cues of the persons and the game context. The iCub will produce an action composed of a game-related action, and affective behavior. The impact of the produced action on the game and the persons is read and used to identify if the action was proper, given the situation, and to modulate the learning of novel actions.

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.00744v1-figure-c44a2d75-6712-473e-a19e-d3e45ee25fa3.jpg)

game, each player receives a set of cards, with different colors, numbers, and possible actions. Starting from a preselected card, the players have to discard a card which matches the color or number of the pre-selected card. The game happens in turns. The first player which discard all the cards at hand wins the game. The action cards help to improve the competitiveness of the game by providing specific gamechange events, such as 'make the next player collect 4 cards', or ' invert the playing order'. In our scenario, the iCub knows the rules of the game and can do a set of pre-defined actions such as check its cards, discarding and collecting cards, calling 'UNO', and passing its turn. The robot can also display certain affective responses, using different body movements and facial expressions.

While playing, the robot will observe the individual affective behavior of the other humans and will decide which gamerelated action to take and which affective expression it will display. The UNO card game was chosen as it is a relatively controllable scenario, where each player has its turn to make an action, a highly competitive game in which trying to identify, or even manipulate, the other players' affective behavior is an important game strategy, and, most importantly, the game provides several turns of context-dependent interaction as illustrated in Figure 2.

The iCub must take into consideration how the three persons are behaving while playing to be able to react specifically and in an appropriate manner to each of them. The appropriate manner will be selected based on different desired targets: the iCub must win the game (maximizing the game strategy), it must improve the engagement of the other players (maximizing the generation of affective behavior), or it must provide a playing behavior which is accepted by the subjects as humanlike (maximizing both affective behavior and game strategies).

As each person expresses emotions in a highly individual manner, the iCub must adapt its perception to each of them. But, as they pose as a group, it also must take into consideration how the affective behavior of each person impacts the reading of the emotional state of the entire group. The iCub must take into consideration the gradual evolution of the affective states of each player within the game, and associate it with the temporal change within the game context.

We derived this scenario from common real-world applications for companion and tutoring robots used in elderly care centers and interaction with children. Our scenario, however, increases the complexity of the interaction by providing an unrestricted turn-taking initiative, where the persons will be instructed to act as natural as possible. To provide a realtime response with the iCub is of much importance, as it must be perceived as an active and autonomous player. Also, the interplay between the individual perception and the group affective behavior is very important for grounding the produced actions and for the developing of different gamerelated strategies.

## III. TOWARDS A COGNITIVE FRAMEWORK

## A. Affective Modeling

Recently, we proposed a novel hybrid neural framework to deal with the problem of adapting deep neural models towards specific characteristics of perceiving emotion expressions in an online manner [2]. A series of pre-trained convolution channels were used to represent facial expressions, body movement, and prosody on the speech from one person. The GWR network received the multimodal representation as input, and learning, online, how to cluster the sensory stimuli into different affective concepts. The GWR adapts to the perceived stimuli by creating or removing neurons and it can learn, quickly, how to model novel information. Each neuron represents a prototype emotional concept, and are used to represent affective expressions. We explored the use of individual GWRs to learn prototype emotional concepts from individual persons and used them as specific affective memories which represent how that specific person expresses emotions [3]. The affective memories were evaluated on an emotion recognition task and were shown to be a state-of-theart model on recognizing spontaneous emotion expressions.

To deal with the proposed scenario, we will take inspirations on the affective memory networks. We must devise a novel affective memory model which can deal with the temporal changes within the interaction, in particular, able to integrate the perceived stimuli asynchronously. The model will process facial expressions, body movement, and speech signals from all persons in the scenario, which are all expressed within different contextual times. To deal with this difference we will investigate the use of gated recurrent units (GRUs) [7] which will contribute to a time-compressed representation of an expression. A general attention mechanism will also important to avoid that intense emotional changes are neutralized by long periods of displaying the same affective behavior. To address this problem we will integrate the concepts of local and global attention to the convolution channels of the model. Also, we will investigate the use of recurrent pooling layers will be used to highlight specific periods of the input stimuli which contribute more to emotion recognition.

To provide a general affective analysis of the situation, we will integrate the individual affective memories into one emotional read. We will investigate the use of recurrent gamma connections [18] within a GWR to model the impact that each person will have in the general emotional representation. The GWR, once again, will provide the model with an online association learning mechanism, which will be able to depict, in an online manner, how both persons are behaving based on their affective information.

It is important to ground the proposed model on existing literature on emotion processing. This allows a direct comparison with humans not only on the performance level but also on behavioral aspects. Thus, the design of the novel learning rules, the topological and conceptual aspects of the model will be inspired by behavioral knowledge on how humans perceived emotion expressions.

## B. Modulating Game Strategies with Social Reactions

To learn how to provide proper behavior, we will develop and evaluate a reinforcement learning model based on actorcritic neural networks. The actor-network will learn how to associate the current group perception state of the robot and the affective memory related to the person that the robot is currently aiming to play with, with a set of game and affective behavior actions.

By perceiving the reaction of the players to the selected action, and the change in the game context, the iCub will maximize the production of a proper reaction. The critic network will learn how to maximize the actor-network output based on the desired target: a reward function based only on maximizing the game score of the iCub will produce a set of behaviors which will drive the iCub to win the game; by providing a set of reaction that maintain the group affective state positive, will direct the iCub to produce behaviors which seem as enjoyable by the other players; and creating a reward function based on both improving the game score of the iCub and maintaining a positive affective behavior on the group, will develop a certain game strategy that we assume it will be perceived as more human-like.

To maximize the learning strategy based on the few data points collected in each interaction, we will investigate the use of predictive stimulation [16] to provide a continual learning mechanism for the actor-critic network. We will collect videos of persons playing the game, and use it to create a replay memory pool, which integrates perceptionaction-reaction associations. Different modulation techniques, in particular, based on the interplay between exploration and exploitation will be devised to avoid that the model overfits towards the memory replay.

## IV. EVALUATION PROTOCOL

We will evaluate the proposed model using two strategies: one offline, which will support the topological development of each model, and one with real persons, which will enforce the real-time processing properties demanded by our scenario.

The offline training will be important for the development of novel learning rules and hyperparameter optimization of the model. We will simulate the game using the strategies collected from the recorded data, and the iCub simulator with a set of pre-defined game strategies. We will evaluate the individual modules of the proposed model, as well as different integration strategies. For this evaluation, objective measures such as the accuracy on recognizing emotion expressions, the concordance between the model and external human observers and the processing time will be maximized.

The real-world evaluation will involve the described scenario. Ideally, we will have multiple sessions with the same participants happening over a month to capture the important characteristics of the affective behavior of each participant. We will shuffle each group of participants during the sessions to avoid perception bias associated always with the same group of participants. The expected behavior is that the robot learns how to play the game based on the three different goals, and devise specific playing characteristics for each person. After each interaction, the participants will fill out different HRIrelated questionnaires. We are in particular interested in the Asch's personality impression questionnaire [1], which will give us the persons assessment of how the robot is behaving, and the Godspeed questionnaire [4], which will help us to understand the persons general opinion about the impact of the robot in the interaction. Combining these results with the objective measures of the offline evaluation will give us a general perspective on how the proposed model impacts the general acceptance of the robot in a competitive scenario.

## V. CONCLUSION

The action-perception cycle of perceiving emotion expressions, and responding with socially accepted actions is one of the most important aspects of human-human natural communication. In this paper, we describe our intentions to develop a hybrid neural framework for learning proper affective responses with an iCub robot in a competitive game scenario. The framework will comprise of two modules, the first one implementing deep and unsupervised neural networks for learning how to represent the group affective behavior from the specific characteristics of individual persons. The second module uses an actor-critic reinforcement learning neural network to provide actions, based on the perceived affective behavior. The impact of a chosen actions, based on three different desired targets, will be used as learning modulation for the model. We also present a series of evaluation strategies on how to provide a balance between objective and subjective analysis to improve the robustness of the model and to assess its impact on our HRI scenario.

## ACKNOWLEDGMENT

The authors gratefully acknowledge partial support from the German Research Foundation DFG under project CML (TRR 169). Alessanda Sciutti was supported by a Starting Grant from the European Research Council (ERC) under the European Unions Horizon 2020 research and innovation programme, G.A. No 804388, wHiSPER.

## REFERENCES

- [1] Solomon E Asch. Forming impressions of personality. The Journal of Abnormal and Social Psychology , 41(3):258, 1946.
- [2] Pablo Barros and Stefan Wermter. Developing crossmodal expression recognition based on a deep neural model. Adaptive behavior , 24(5):373-396, 2016.
- [3] Pablo Barros and Stefan Wermter. A self-organizing model for affective memory. In Neural Networks (IJCNN), 2017 International Joint Conference on , pages 31-38. IEEE, 2017.
- [4] Christoph Bartneck, Dana Kuli´ c, Elizabeth Croft, and Susana Zoghbi. Measurement instruments for the anthropomorphism, animacy, likeability, perceived intelligence, and perceived safety of robots. International journal of social robotics , 1(1):71-81, 2009.
- [5] Balasubramaniyan Chandrasekaran and James M Conrad. Human-robot collaboration: A survey. In SoutheastCon 2015 , pages 1-8. IEEE, 2015.
- [6] Luefeng Chen, Mengtian Zhou, Wanjuan Su, Min Wu, Jinhua She, and Kaoru Hirota. Softmax regression based deep sparse autoencoder network for facial emotion recognition in human-robot interaction. Information Sciences , 428:49-61, 2018.
- [7] Kyunghyun Cho, Bart Van Merri¨ enboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. Learning phrase representations using rnn encoder-decoder for statistical machine translation. arXiv preprint arXiv:1406.1078 , 2014.
- [8] Nikhil Churamani, Pablo Barros, Erik Strahl, and Stefan Wermter. Learning empathy-driven emotion expressions using affective modulations. In 2018 International Joint Conference on Neural Networks (IJCNN) , pages 1-8. IEEE, 2018.
- [9] Molly K Crossman, Alan E Kazdin, and Elizabeth R Kitt. The influence of a socially assistive robot on mood, anxiety, and arousal in children. Professional Psychology: Research and Practice , 49(1):48, 2018.
- [10] Ning Hao, Hua Xue, Huan Yuan, Qing Wang, and Mark A Runco. Enhancing creativity: Proper body posture meets proper emotion. Acta psychologica , 173:32-40, 2017.
- [11] Elaine Hatfield, John T Cacioppo, and Richard L Rapson. Emotional contagion. Current directions in psychological science , 2(3):96-100, 1993.
- [12] Masakazu Hirokawa, Atsushi Funahashi, Yasushi Itoh, and Kenji Suzuki. Adaptive behavior acquisition of a robot based on affective feedback and improvised teleoperation. IEEE Transactions on Cognitive and Developmental Systems , 2018.
- [13] Peter J Lang, Margaret M Bradley, Bruce N Cuthbert, et al. Motivated attention: Affect, activation, and action. Attention and orienting: Sensory and motivational processes , 97:135, 1997.
- [14] Iolanda Leite, Carlos Martinho, and Ana Paiva. Social robots for longterm interaction: a survey. International Journal of Social Robotics , 5(2):291-308, 2013.
- [15] Giorgio Metta, Giulio Sandini, David Vernon, Lorenzo Natale, and Francesco Nori. The icub humanoid robot: an open platform for research in embodied cognition. In Proceedings of the 8th workshop on performance metrics for intelligent systems , pages 50-56. ACM, 2008.
- [16] Yukie Nagai. Predictive learning: its key role in early cognitive development. Philosophical Transactions of the Royal Society B , 374(1771):20180030, 2019.
- [17] Kevin N Ochsner and James J Gross. Cognitive emotion regulation: Insights from social cognitive and affective neuroscience. Current directions in psychological science , 17(2):153-158, 2008.
- [18] G.I. Parisi, J. Tani, C. Weber, and S Wermter. Lifelong learning of spatiotemporal representations with dual-memory recurrent selforganization. In arXiv:1805.10966 , 2018.
- [19] Evangelos Sariyanidi, Hatice Gunes, and Andrea Cavallaro. Automatic analysis of facial affect: A survey of registration, representation, and recognition. IEEE transactions on pattern analysis and machine intelligence , 37(6):1113-1133, 2014.
- [20] Candace L Sidner, Cory D Kidd, Christopher Lee, and Neal Lesh. Where to look: a study of human-robot engagement. In Proceedings of the 9th international conference on Intelligent user interfaces , pages 7884. ACM, 2004.
- [21] Nguyen Tan Viet Tuyen, Sungmoon Jeong, and Nak Young Chong. Emotional bodily expressions for culturally competent robots through long term human-robot interaction. In 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) , pages 2008-2013. IEEE, 2018.
- [22] Greet Van de Perre, Hoang-Long Cao, Albert De Beir, Pablo G´ omez Esteban, Dirk Lefeber, and Bram Vanderborght. Generic method for generating blended gestures and affective functional behaviors for social robots. Autonomous Robots , 42(3):569-580, 2018.
Pablo Barros Stefan Wermter Alessandra Sciutti
Towards Learning How to Properly Play UNO with the iCub Robot
While interacting with another person, our reactions and behavior are much
affected by the emotional changes within the temporal context of the
interaction. Our intrinsic affective appraisal comprising perception,
self-assessment, and the affective memories with similar social experiences
will drive specific, and in most cases addressed as proper, reactions within
the interaction. This paper proposes the roadmap for the development of
multimodal research which aims to empower a robot with the capability to
provide proper social responses in a Human-Robot Interaction (HRI) scenario.