Body: <p>I have an unusual but very interesting problem. I have a game that is very similar to Toon Blast (a puzzle mobile game). It's based on a Match-2 mechanic in which you can destroy 2 or more connected blocks and your goal is to complete all the required objectives (collect X color blocks, destroy 30 balloons, etc).</p>
  <p>I have tons of levels and the ML solver seems to perform very well for all kinds of obstacles - except Quicksand.</p>
  <p>Quicksand is a special object that replicates itself to a nearest tile whenever a user makes a move. If the user destroyed a quicksand in his turn, a quicksand won't be replicated. So basically the fastest way to destroy quicksand is to make sure you destroy as much quicksand as you can in each turn so it won't replicate and cover your board.</p>
  <p>I use ML-Agents from Unity (<a href="https://github.com/Unity-Technologies/ml-agents" rel="nofollow noreferrer">https://github.com/Unity-Technologies/ml-agents</a>) and I just give the agent <code>reward=1f</code> whenever it completes an objective (destroy 1 obstacle) and I <code>subtract 1f</code> from reward whenever it performs a move.</p>
  <p>For simpler non-replicating obstacles it works perfectly. For example, you click 2 blocks next to a balloon - it will pop a balloon, add 1f as a reward and at the same time remove 1f for using a move.</p>
  <p>This way the agent learns to make as few moves as possible.</p>
  <p>Below you'll find how the Quicksand works and some simple obstacle - Balloon.</p>
  <p><strong>Quicksand preview</strong> (sorry for bad quality, 2mb max size)</p>
  <p><a href="https://i.sstatic.net/NS1Bn.gif" rel="nofollow noreferrer"><img src="https://i.sstatic.net/NS1Bn.gif" alt="enter image description here" /></a></p>
  <p><strong>Balloon preview</strong></p>
  <p><a href="https://i.sstatic.net/bJGLW.gif" rel="nofollow noreferrer"><img src="https://i.sstatic.net/bJGLW.gif" alt="enter image description here" /></a></p>
  <p>My issue is that no matter what, I can't teach it to solve quicksand. With the above rewarding approach, I think this strange creature learned that by actually REPLICATING quicksand it's gaining more reward because it can destroy it later (actually it's not because moves give -1f so it's more-less equal).</p>
  <p>I've tried not giving reward for the quicksand, so it only loses rewards by using moves. But it doesn't work either, I'm not sure why.</p>
  <p>Do you guys have any idea how this kind of things should be taught?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-12-20T08:49:56.060Z
Comments.Text: Do you know which RL are you using under the hood with ml-agents? It may be a good idea to investigate that if you don't know it. Having said that, if I understand correctly, you have 2 distinct environments: Ballon and Quicksand. If the objective in these environments is different, you should probably change the reward function, but it seems that you're using the same reward function for both. Right?
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-12-20T21:54:42.297Z
Comments.Text: I wonder if it would help to make the reward a function of the *change* in obstacle tile count following the move. So allowing an obstacle tile to replicate is punished more severely than simply neglecting a non-replicating obstacle.
Comments.UserDisplayName: DMGregory
Tags: <machine-learning><reinforcement-learning><game-ai><reward-functions><reward-design>
Title: How to teach Machine Learning Agent to destroy replicating objects in a puzzle game?
UserDisplayName: Jacob
ThreadText: # How to teach Machine Learning Agent to destroy replicating objects in a puzzle game?
  
  (Asked by: username_0 on 2021-12-18)
  
  I have an unusual but very interesting problem. I have a game that is very similar to Toon Blast (a puzzle mobile game). It's based on a Match-2 mechanic in which you can destroy 2 or more connected blocks and your goal is to complete all the required objectives (collect X color blocks, destroy 30 balloons, etc).
  
  I have tons of levels and the ML solver seems to perform very well for all kinds of obstacles - except Quicksand.
  
  Quicksand is a special object that replicates itself to a nearest tile whenever a user makes a move. If the user destroyed a quicksand in his turn, a quicksand won't be replicated. So basically the fastest way to destroy quicksand is to make sure you destroy as much quicksand as you can in each turn so it won't replicate and cover your board.
  
  I use ML-Agents from Unity (<https://github.com/Unity-Technologies/ml-agents>) and I just give the agent `reward=1f` whenever it completes an objective (destroy 1 obstacle) and I `subtract 1f` from reward whenever it performs a move.
  
  For simpler non-replicating obstacles it works perfectly. For example, you click 2 blocks next to a balloon - it will pop a balloon, add 1f as a reward and at the same time remove 1f for using a move.
  
  This way the agent learns to make as few moves as possible.
  
  Below you'll find how the Quicksand works and some simple obstacle - Balloon.
  
  **Quicksand preview** (sorry for bad quality, 2mb max size)
  
  [![enter image description here](https://i.sstatic.net/NS1Bn.gif)](https://i.sstatic.net/NS1Bn.gif)
  
  **Balloon preview**
  
  [![enter image description here](https://i.sstatic.net/bJGLW.gif)](https://i.sstatic.net/bJGLW.gif)
  
  My issue is that no matter what, I can't teach it to solve quicksand. With the above rewarding approach, I think this strange creature learned that by actually REPLICATING quicksand it's gaining more reward because it can destroy it later (actually it's not because moves give -1f so it's more-less equal).
  
  I've tried not giving reward for the quicksand, so it only loses rewards by using moves. But it doesn't work either, I'm not sure why.
  
  Do you guys have any idea how this kind of things should be taught?
  
  **Tags:** machine-learning, reinforcement-learning, game-ai, reward-functions, reward-design
  
  > **Comments:**
  > **username_1**: Do you know which RL are you using under the hood with ml-agents? It may be a good idea to investigate that if you don't know it. Having said that, if I understand correctly, you have 2 distinct environments: Ballon and Quicksand. If the objective in these environments is different, you should probably change the reward function, but it seems that you're using the same reward function for both. Right?  
  > **username_2**: I wonder if it would help to make the reward a function of the *change* in obstacle tile count following the move. So allowing an obstacle tile to replicate is punished more severely than simply neglecting a non-replicating obstacle.  
  
  ---
  
  ## 0 answers
  
  