Dynamic Safe Interruptibility for Decentralized Multi-Agent Reinforcement Learning
In reinforcement learning, agents learn by performing actions and observing their outcomes. Sometimes, it is desirable for a human operator to interrupt an agent in order to prevent dangerous situations from happening. Yet, as part of their learning process, agents may link these interruptions, that impact their reward, to specific states and deliberately avoid them. The situation is particularly challenging in a multi-agent context because agents might not only learn from their own past interruptions, but also from those of other agents. Orseau and Armstrong [16] defined safe interruptibility for one learner, but their work does not naturally extend to multi-agent systems. This paper introduces dynamic safe interruptibility, an alternative definition more suited to decentralized learning problems, and studies this notion in two learning frameworks: joint action learners and independent learners. We give realistic sufficient conditions on the learning algorithm to enable dynamic safe interruptibility in the case of joint action learners, yet show that these conditions are not sufficient for independent learners. We show however that if agents can detect interruptions, it is possible to prune the observations to ensure dynamic safe interruptibility even for independent learners.
This paper presents an extension of the safe interruptibility (SInt) 
framework to the multi-agent case. The authors argue that the 
original definition of safe interruptibility is difficult to use in 
this case and give a more constrained/informed one called 'dynamic safe 
interruptibility' (DSInt) based on whether the update rule depends 
on the interruption probability. The joint action case is considered 
first and it is shown that DSInt can be achieved. The case of 
independent learners is then considered, with a first result showing 
that independent Q-learners do not satisfy the conditions of the 
definition of DSInt. The authors finally propose a model where the 
agents are aware of each others interruptions, and interrupted 
observations are pruned from the sequence, and claim that this model 
verify the definition of DSInt.
The paper is mostly well-written, well motivated, offers novel ideas 
and appears mostly technically correct. The level of formalism is 
good, emphasizing on the ideas rather than rendering a boring 
sequence of symbols, but this also hurts somewhat the proof reading 
of the appendix, so I can't be 100% confident about the results.
* Main comments:
- What is the relation between the authors' definition of DSInt and 
the original version? In particular, do we have DSInt implies SInt, 
or the converse, or neither?
- In the proof of Lemma 1, p. 12, third equality, a condition on 'a' 
is missing in the second P term. As far as I understand, this 
condition cannot be removed, preventing this term from being pulled 
out of the sum.
* Minor comments:
- The motivating example is pretty good. On a side note however, it 
seems that in this scenario the knowledge of others' interruptions 
may still be inadequate, as in the general case the different 
vehicles may not even be of the same brand and thus may not exchange 
information (unless some global norm is defined?). This is not so 
much of a criticism, as one needs to start somewhere, but a 
discussion about that could go in the conclusion.
- Definition 2: This should define "dynamic safe interruptibility", 
not just "safe interruptibility" which is already defined in [16].
- The notation used for open intervals is at times standard (in 
the appendix) and at other times non-standard (as in the main text). 
This should be made homogeneous (preferably using the standard 
notation).
- Def 5: "multi-agent systems" -> "multi-agent system"
- Proof of Thm 2: On the first equality in the display, we lost the 
index (i). 
- The second line after the display should probably read  
\hat{Q^m_t} instead of Q^{(m)}_t.
- Missing period at the end of the proof.
- Proof of Thm 3: "two a and b" -> "two agents a and b" (or learner, 
player?)
- \gamma is not defined AFAICT
- Proof of Thm 4, "same than" -> "same as"
- Does it mean that independent learners are DSInt but not SInt?
- Penultimate sentence of the conclusion, "amount exploration" -> +of
- p.12 after Lemma 2 "each agents do not" -> "each agent does" or 
"no agent learns"
- last line of the display in the proof of Lemma 1: Use the \left[ 
and \right]  commands (assuming LaTeX is being used) to format 
properly.
- proof of Thm 4: q is undefined.