Answers.Body: <p>Firstly contrary to your claim, sometimes it's a requirement for artificial agents to choose multiple different goals for the sake of an intended goal such as <a href="https://en.wikipedia.org/wiki/Deep_reinforcement_learning#Goal-conditioned_reinforcement_learning" rel="nofollow noreferrer">goal conditioned RL</a>.</p>
  <blockquote>
  <p>Another active area of research is in learning goal-conditioned policies, also called contextual or universal policies that take in an additional goal as input to communicate a desired aim to the agent.</p>
  </blockquote>
  <p>Bostrom further explored the idea of <a href="https://en.wikipedia.org/wiki/Instrumental_convergence" rel="nofollow noreferrer">instrumental convergence</a> that intelligent agents might pursue certain instrumental goals regardless of their final objectives in his paper <em>Motivation and Instrumental Rationality in Advanced Artificial Agents</em>.</p>
  <p>As for your RCT which is not a game theory but a classic evidential decision theory, the extant RL in AI is consistent with RCT and may be viewed as an advanced sequential multi-shot version of RCT. S. Russell discussed the relationship between rationality and intelligence in his <em>Rationality and Intelligence: A Brief Update</em>, offering insights that may align with your interest in AI application issue of RCT.</p>
  <blockquote>
  <p>The concept of rational agency has long been considered a leading candidate to fulfill this role. This paper, which updates a much earlier version (Russell, 1997), reviews the sequence of conceptual shifts leading to a different candidate, <em>bounded optimality</em>, that is closer to our informal conception of intelligence and reduces the gap between theory and practice.</p>
  </blockquote>
  <p>Hadfield-Menell et al. presents a game theory and RCT consistent formal model of an AI system that includes an off-switch controlled by humans exploring how to design AI systems that can be safely interrupted in their paper <em>The Off-Switch Game</em>.</p>
  <blockquote>
  <p>It is clear that one of the primary tools we can use to mitigate the potential risk from a misbehaving AI system is the ability to turn the system off. As the capabilities of AI systems improve, it is important to ensure that such systems do not adopt subgoals that prevent a human from switching them off. This is a challenge because many formulations of rational agents create strong incentives for self-preservation... Our key insight is that for R to want to preserve its off switch, it needs to be uncertain about the utility associated with the outcome, and to treat H’s actions as important observations about that utility... We conclude that giving machines an appropriate level of uncertainty about their objectives leads to safer
  designs, and we argue that this setting is a useful generalization of the classical AI paradigm of rational agents.</p>
  </blockquote>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-08-04T00:54:26.853Z
Answers.LastActivityDate: 2024-08-04T00:54:26.853Z
Answers.UserDisplayName: cinch
Answers.Body: <p>@raving-bandit, the common argument on your first question is that an AI system may take its goal to the extreme and divert all the world's resources to it, causing human extinction (see e.g. here: <a href="https://cepr.org/voxeu/columns/ai-and-paperclip-problem" rel="nofollow noreferrer">https://cepr.org/voxeu/columns/ai-and-paperclip-problem</a>)
  RCT may not be the best framework because we typically assume everyone knows the game being played, whereas an AI system is dangerous because it could think of moves no-one would expect. This also relates to your second question. It appears that epistemic game theory (<a href="https://plato.stanford.edu/entries/epistemic-game/" rel="nofollow noreferrer">https://plato.stanford.edu/entries/epistemic-game/</a>) or limited rationality models like cognitive hierarchies (<a href="https://en.wikipedia.org/wiki/Cognitive_hierarchy_theory" rel="nofollow noreferrer">https://en.wikipedia.org/wiki/Cognitive_hierarchy_theory</a>) could be a better way to approach the problem than RCT.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-08-03T15:36:44.387Z
Answers.LastActivityDate: 2024-08-03T15:36:44.387Z
Answers.UserDisplayName: Arseniy  Samsonov
Body: <p>My background is in political economy and game theory. I am interested in the discussion on AI risk and alignment, but I have so far failed to find work on this that seriously engages with classic axiomatic rational choice theory (RCT).</p>
  <p>Some claims are often made that, to me, contradict the basic tenets of RCT. For example, it is often suggested that a sufficiently advanced AGI might come up with goals of its own that put at risk humanity. However, according to RCT, agents can only choose <em>actions</em>, whereas their preferences (goals) are primitives of the model. It is nonsensical to think of an agent as choosing his goals, except in some well-defined dynamic model (say, Becker's model of rational addiction) where the future preferences result from actions that can be derived, by equilibrium analysis, from primitive preferences.</p>
  <p>Another claim that perplexes me is that the actions of a sufficiently intelligent AI may be impossible to predict. However, to me the more intelligent an agent, the more its actions should approximate those of an expected utility maximizer.</p>
  <p>I wonder if serious people are working on this from a different axiomatic system. What are some good sources (at any level of technicality) that I should read? Better if books.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-11-28T18:21:28.040Z
Comments.Text: Human Compatible by Stuart Russell?
Comments.UserDisplayName: Jaume Oliver Lafont
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-11-28T20:44:36.470Z
Comments.Text: I'm looking for something a bit more rigorous: something structured as Definition-Lemma-Theorem, in the tradition of rational choice / game theory.
Comments.UserDisplayName: raving-bandit
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-08-03T15:39:20.110Z
Comments.Text: @raving-bandit I have the same background as you and am interested in similar questions, which is how I found your post; it would be amazing to connect and bounce ideas. I call on the moderators not to remove this post since I would like to contact the OP for potential research collaboration, but I am unsure how to do it otherwise.
Comments.UserDisplayName: Arseniy  Samsonov
Tags: <game-theory><value-alignment><risk-management>
Title: Is there serious game-theoretic work on AI risk and alignment?
UserDisplayName: raving-bandit
ThreadText: # Is there serious game-theoretic work on AI risk and alignment?
  
  (Asked by: username_0 on 2023-11-28)
  
  My background is in political economy and game theory. I am interested in the discussion on AI risk and alignment, but I have so far failed to find work on this that seriously engages with classic axiomatic rational choice theory (RCT).
  
  Some claims are often made that, to me, contradict the basic tenets of RCT. For example, it is often suggested that a sufficiently advanced AGI might come up with goals of its own that put at risk humanity. However, according to RCT, agents can only choose *actions*, whereas their preferences (goals) are primitives of the model. It is nonsensical to think of an agent as choosing his goals, except in some well-defined dynamic model (say, Becker's model of rational addiction) where the future preferences result from actions that can be derived, by equilibrium analysis, from primitive preferences.
  
  Another claim that perplexes me is that the actions of a sufficiently intelligent AI may be impossible to predict. However, to me the more intelligent an agent, the more its actions should approximate those of an expected utility maximizer.
  
  I wonder if serious people are working on this from a different axiomatic system. What are some good sources (at any level of technicality) that I should read? Better if books.
  
  **Tags:** game-theory, value-alignment, risk-management
  > **Comments:**
  > **username_1**: Human Compatible by Stuart Russell?  
  > **username_0**: I'm looking for something a bit more rigorous: something structured as Definition-Lemma-Theorem, in the tradition of rational choice / game theory.  
  > **username_2**: @raving-bandit I have the same background as you and am interested in similar questions, which is how I found your post; it would be amazing to connect and bounce ideas. I call on the moderators not to remove this post since I would like to contact the OP for potential research collaboration, but I am unsure how to do it otherwise.  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_3 on 2024-08-04. Score: 0
  
  Firstly contrary to your claim, sometimes it's a requirement for artificial agents to choose multiple different goals for the sake of an intended goal such as [goal conditioned RL](https://en.wikipedia.org/wiki/Deep_reinforcement_learning#Goal-conditioned_reinforcement_learning).
  
  > Another active area of research is in learning goal-conditioned policies, also called contextual or universal policies that take in an additional goal as input to communicate a desired aim to the agent.
  
  Bostrom further explored the idea of [instrumental convergence](https://en.wikipedia.org/wiki/Instrumental_convergence) that intelligent agents might pursue certain instrumental goals regardless of their final objectives in his paper *Motivation and Instrumental Rationality in Advanced Artificial Agents*.
  
  As for your RCT which is not a game theory but a classic evidential decision theory, the extant RL in AI is consistent with RCT and may be viewed as an advanced sequential multi-shot version of RCT. S. Russell discussed the relationship between rationality and intelligence in his *Rationality and Intelligence: A Brief Update*, offering insights that may align with your interest in AI application issue of RCT.
  
  > The concept of rational agency has long been considered a leading candidate to fulfill this role. This paper, which updates a much earlier version (Russell, 1997), reviews the sequence of conceptual shifts leading to a different candidate, *bounded optimality*, that is closer to our informal conception of intelligence and reduces the gap between theory and practice.
  
  Hadfield-Menell et al. presents a game theory and RCT consistent formal model of an AI system that includes an off-switch controlled by humans exploring how to design AI systems that can be safely interrupted in their paper *The Off-Switch Game*.
  
  > It is clear that one of the primary tools we can use to mitigate the potential risk from a misbehaving AI system is the ability to turn the system off. As the capabilities of AI systems improve, it is important to ensure that such systems do not adopt subgoals that prevent a human from switching them off. This is a challenge because many formulations of rational agents create strong incentives for self-preservation... Our key insight is that for R to want to preserve its off switch, it needs to be uncertain about the utility associated with the outcome, and to treat H’s actions as important observations about that utility... We conclude that giving machines an appropriate level of uncertainty about their objectives leads to safer
  > designs, and we argue that this setting is a useful generalization of the classical AI paradigm of rational agents.
  
  
  ---
  
  ### Answer by username_2 on 2024-08-03. Score: 0
  
  @raving-bandit, the common argument on your first question is that an AI system may take its goal to the extreme and divert all the world's resources to it, causing human extinction (see e.g. here: <https://cepr.org/voxeu/columns/ai-and-paperclip-problem>)
  RCT may not be the best framework because we typically assume everyone knows the game being played, whereas an AI system is dangerous because it could think of moves no-one would expect. This also relates to your second question. It appears that epistemic game theory (<https://plato.stanford.edu/entries/epistemic-game/>) or limited rationality models like cognitive hierarchies (<https://en.wikipedia.org/wiki/Cognitive_hierarchy_theory>) could be a better way to approach the problem than RCT.
  
  
  ---
  
  