Answers.Body: <p>The simplest thing to do when you make you first implementation of the agent, is to automate decisions like this, in order to keep representations and decisions simple.</p>
  
  <p>However, if you want to explore tactics surrounding declaration, then I think the following applies:</p>
  
  <ul>
  <li><p>There should be an initial round of actions where the agent may get to decide whether or not to declare a Ronda, based on the cards it holds. These will be different action choices to playing cards, so you would need to alter your action representation to include those choices. Only allow action choices which are valid, so if it is not valid to declare a Ronda or a Tringa when a player does not have one, then the player does not get to make that choice.</p></li>
  <li><p>You may want to add a state feature "has a Ronda" and "has a Tringa" for the agent's player, to help with the action decision. </p></li>
  <li><p>You <em>should</em> also add a state feature for each player according to whether they <em>declared</em> a Ronda or a Tringa.</p></li>
  <li><p>Rather than have the agent learn to detect a lie, given your comment that all cards are played so it is easy to tell (there are only 8 cards total in play by the end), then I would just assume lies are automatically found out and include that in the game engine. In other words, the penalty is always paid.</p></li>
  <li><p>The interesting question is then whether withholding the declaration can give a tactical advantage when the round is being played (because your opponent knows less about your hand), and whether that advantage offsets the inevitable penalty. This might not be true in your card game, but could be true in games with similar choices.</p></li>
  </ul>
  
  <blockquote>
    <p>Could a DQN handle this?</p>
  </blockquote>
  
  <p>A DQN is maybe going to struggle with partial information in this game. The opponent's cards are hidden from the network, but could have a non-random influence on the opponent's choices of action. It is possible that you will need to investigate agents that can solve <a href="https://en.wikipedia.org/wiki/Partially_observable_Markov_decision_process" rel="nofollow noreferrer">POMDPs</a> to get the best player. </p>
  
  <p>I don't know for certain though. It depends on how much tactical advantage there is in having concealed cards, or how much that is just luck that plays out much the same whether you know the opponent's cards or not. If there is strategy, and ways to determine/guess what the opponent holds based on your cards and their actions so far, this is more like a POMDP.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-07-08T21:00:48.060Z
Answers.Text: Thanks for the detailed answer, I appreciate the informations you provided! I guess as a first step I will try to make the agent know when he has a Ronda (given the cards in his hand as features) and to announce it as an action, I will leave detecting opponent lies or lying itself to a "version 2" of the side project. Thank you.
Answers.UserDisplayName: Haytam
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-07-08T20:47:56.393Z
Answers.LastActivityDate: 2018-07-08T20:47:56.393Z
Answers.UserDisplayName: Neil Slater
Body: <p>More informations on the card game I'm talking about are in my last question here: <a href="https://ai.stackexchange.com/questions/7049/dqn-input-representation-for-a-card-game">DQN input representation for a card game</a></p>
  
  <p>So I was thinking about the output of the q neural network and, aside from which card to play, I was wondering if the agent can announce things.</p>
  
  <p>Imagine you have the current hand: <code>2, 4, 11, 2</code> (The twos are different card type).<br>
  When you're playing the game and you get dealt a hand like this, you have to announce that you have the same number twice (called Ronda) or thrice (called Tringa) before anyone plays a card on the table. Lying about it gets you a penalty.</p>
  
  <p>Could a DQN handle this? I don't know if adding "Announcing a Ronda/Tringa" as an action would actually help. I mean, can this be modeled for the  NN or should I just automate this and spare the agent having to announce it everytime.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2018-07-08T19:44:24.840Z
Comments.Text: Is there any possible tactical advantage to lying and accepting the penalty, in the normal game?
Comments.UserDisplayName: Neil Slater
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2018-07-08T20:11:30.753Z
Comments.Text: Thing is, if both players have a Ronda, the one with the highest number wins 2 points so u could lie and he gets only 1 point but thats too human i believe, plus if they catch u lying u get the penalty either way so im starting to belive that i shouldnt include it?
Comments.UserDisplayName: Haytam
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2018-07-08T20:15:03.320Z
Comments.Text: If lying/concealing the Ronda, plus detecting or calling out the lies is part of the game, then I think it is up to you when implementing the bot. Perhaps save it for v2 if you are just starting out with first implementation. How hard in reality is it to detect the lie? If it is something that will always be noticed by a perceptive player with a good memory, then perhaps it is not so good in the automated game, because it may be possible to fully automate it without any real use of AI in the first place.
Comments.UserDisplayName: Neil Slater
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2018-07-08T20:16:59.303Z
Comments.Text: It's not that hard to detect that a player lied about a Ronda, since you only have to remember the 4 cards he played and you'll find out.
Comments.UserDisplayName: Haytam
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2018-07-08T20:19:18.580Z
Comments.Text: In that case it seems to be a minor tactic in any game with experienced/good players, and you could maybe skip it. However, it is an interesting question in general - other card games may not have such a clear decision about when to include this kind of tactic.
Comments.UserDisplayName: Neil Slater
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2018-07-08T20:22:45.320Z
Comments.Text: The thing is, in order to catch your opponent lying you have to wait for him to play all his 4 cards, and I cannot think about how would the NN understand that...
Comments.UserDisplayName: Haytam
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2018-07-08T20:26:26.070Z
Comments.Text: Let us [continue this discussion in chat](https://chat.stackexchange.com/rooms/79905/discussion-between-haytam-and-neil-slater).
Comments.UserDisplayName: Haytam
Tags: <reinforcement-learning><dqn>
Title: Can DQN announce it has things in its hand in a card game?
UserDisplayName: Haytam
ThreadText: # Can DQN announce it has things in its hand in a card game?
  
  (Asked by: username_0 on 2018-07-08)
  
  More informations on the card game I'm talking about are in my last question here: [DQN input representation for a card game](https://ai.stackexchange.com/questions/7049/dqn-input-representation-for-a-card-game)
  
  So I was thinking about the output of the q neural network and, aside from which card to play, I was wondering if the agent can announce things.
  
  Imagine you have the current hand: `2, 4, 11, 2` (The twos are different card type).  
  When you're playing the game and you get dealt a hand like this, you have to announce that you have the same number twice (called Ronda) or thrice (called Tringa) before anyone plays a card on the table. Lying about it gets you a penalty.
  
  Could a DQN handle this? I don't know if adding "Announcing a Ronda/Tringa" as an action would actually help. I mean, can this be modeled for the NN or should I just automate this and spare the agent having to announce it everytime.
  
  **Tags:** reinforcement-learning, dqn
  > **Comments:**
  > **username_1**: Is there any possible tactical advantage to lying and accepting the penalty, in the normal game?  
  > **username_0**: Thing is, if both players have a Ronda, the one with the highest number wins 2 points so u could lie and he gets only 1 point but thats too human i believe, plus if they catch u lying u get the penalty either way so im starting to belive that i shouldnt include it?  
  > **username_1**: If lying/concealing the Ronda, plus detecting or calling out the lies is part of the game, then I think it is up to you when implementing the bot. Perhaps save it for v2 if you are just starting out with first implementation. How hard in reality is it to detect the lie? If it is something that will always be noticed by a perceptive player with a good memory, then perhaps it is not so good in the automated game, because it may be possible to fully automate it without any real use of AI in the first place.  
  > **username_0**: It's not that hard to detect that a player lied about a Ronda, since you only have to remember the 4 cards he played and you'll find out.  
  > **username_1**: In that case it seems to be a minor tactic in any game with experienced/good players, and you could maybe skip it. However, it is an interesting question in general - other card games may not have such a clear decision about when to include this kind of tactic.  
  > **username_0**: The thing is, in order to catch your opponent lying you have to wait for him to play all his 4 cards, and I cannot think about how would the NN understand that...  
  > **username_0**: Let us [continue this discussion in chat](https://chat.stackexchange.com/rooms/79905/discussion-between-haytam-and-neil-slater).  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2018-07-08. Score: 4 | Accepted answer
  
  The simplest thing to do when you make you first implementation of the agent, is to automate decisions like this, in order to keep representations and decisions simple.
  
  However, if you want to explore tactics surrounding declaration, then I think the following applies:
  
  * There should be an initial round of actions where the agent may get to decide whether or not to declare a Ronda, based on the cards it holds. These will be different action choices to playing cards, so you would need to alter your action representation to include those choices. Only allow action choices which are valid, so if it is not valid to declare a Ronda or a Tringa when a player does not have one, then the player does not get to make that choice.
  * You may want to add a state feature "has a Ronda" and "has a Tringa" for the agent's player, to help with the action decision.
  * You *should* also add a state feature for each player according to whether they *declared* a Ronda or a Tringa.
  * Rather than have the agent learn to detect a lie, given your comment that all cards are played so it is easy to tell (there are only 8 cards total in play by the end), then I would just assume lies are automatically found out and include that in the game engine. In other words, the penalty is always paid.
  * The interesting question is then whether withholding the declaration can give a tactical advantage when the round is being played (because your opponent knows less about your hand), and whether that advantage offsets the inevitable penalty. This might not be true in your card game, but could be true in games with similar choices.
  
  > Could a DQN handle this?
  
  A DQN is maybe going to struggle with partial information in this game. The opponent's cards are hidden from the network, but could have a non-random influence on the opponent's choices of action. It is possible that you will need to investigate agents that can solve [POMDPs](https://en.wikipedia.org/wiki/Partially_observable_Markov_decision_process) to get the best player.
  
  I don't know for certain though. It depends on how much tactical advantage there is in having concealed cards, or how much that is just luck that plays out much the same whether you know the opponent's cards or not. If there is strategy, and ways to determine/guess what the opponent holds based on your cards and their actions so far, this is more like a POMDP.
  
  > **Comments:**
  > **username_0**: Thanks for the detailed answer, I appreciate the informations you provided! I guess as a first step I will try to make the agent know when he has a Ronda (given the cards in his hand as features) and to announce it as an action, I will leave detecting opponent lies or lying itself to a "version 2" of the side project. Thank you.  
  
  ---
  
  