Answers.Body: <p>I skimmed through your question and understood that the state/action space is finite, so in this case, RL would be a good option for storage.</p>
  
  <p>The most basic RL technique will keep track of a matrix <strong>Q</strong> &in; &#8477;<sup><em>s</em>&times;<em>a</em></sup>, where <em>s</em> is number of possible states, and <em>a</em> is number of possible actions. </p>
  
  <p>In addition to a small overhead of agent's parameters:  </p>
  
  <ul>
  <li>&alpha; &in; [0,1] learning parameter</li>
  <li>&gamma; &in; [0,1] discount rate, </li>
  <li>&epsilon; &in; [0,1]  exploration rate</li>
  </ul>
  
  <p>If you have a total of 81 states and 10 actions, then your storage would be in the neighborhood of 850 double words.</p>
  
  <p>If your AI will track different states per location, e.g. possible values 0 to 9 + a null, then your state space will grow to 11 * 81 = 968 and the final value matrix will be <strong>Q</strong> &in; &#8477;<sup>968&times;10</sup> &asymp; 20Kb.</p>
  
  <h2>Update</h2>
  
  <p>I am terrible sorry, I made a terrible mistake in my calculations.
  if you are to keep track of 11 possible values in 81 locations, then the correct number should be 11<sup>81</sup>&times; double word size, that is a big number. </p>
  
  <p>Of course, if this is Sudoku and you  and you are not considering illegal moves as possible states, then definitely this will be a way smaller number, but you will still keep track of an action space that is as big as the state space.</p>
  
  <p>You might need to use function approximation instead, or some sort of action state space compression.</p>
  
  <p>sorry </p>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-08-23T19:59:27.930Z
Answers.Text: Don't apologize!  I really appreciate you taking the time, and this answer is helpful.  Really it's only 9 values over 81 positions, but the game mechanics result with endgame states with ~10 dead cells where no integers can be legally played.  Because of the tractable endgame, we don't need to even go that deep, and merely track openings.
Answers.UserDisplayName: DukeZhou
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-08-23T20:01:22.390Z
Answers.Text: PS I added a slight edit to your first suggestion to the description for the reinforcement-learning tag.  Thanks for getting involved.  If you want to discuss further tweaks, feel free to open a chat, and invite me from a comment here.
Answers.UserDisplayName: DukeZhou
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-08-23T13:38:29.403Z
Answers.LastActivityDate: 2017-08-23T19:24:52.463Z
Answers.LastEditDate: 2017-08-23T19:24:52.463Z
Answers.UserDisplayName: A.Rashad
Body: <p>This concerns a set of finite, non-trivial, combinatorial games <a href="http://www.fundamentalcombinatronics.com/rules-of-m/" rel="nofollow noreferrer">[M]</a> in the form of an app. <a href="http://www.mclassgames.com/powerofm/" rel="nofollow noreferrer">A sample game can be found here</a>. </p>
  
  <p>Because this is a mass market product, we can't take up too much space, and the AI needs to be able to run locally since connectivity cannot be assumed. The current size of the Android kernel is &lt; 7MB.</p>
  
  <p>The goal is not sheer AI strength, but respectable AI strength, sufficient to beat the above-average human player. <em>(The current strongest, weak automata, using a few heuristics, is already capable of beating the average human player.)</em> </p>
  
  <p>Because the games are finite, the gametrees eventually become tractable, allowing for perfect endgames.  Resource stealing strategies and trap-avoidance can also be effected with shallow look-ahead at all phases, and the patterns are much easier for automata to recognize than for humans.</p>
  
  <p><strong>In this context, reinforcement would be mostly utilized to "tune the automata to the style of the human player," and produce different automata on different devices, which could subsequently play against each other as proxies for their human partners.</strong></p>
  
  <p>The game data can be stored efficiently, initially requiring only 2 bytes per position (a value 0-9 and a coordinate 1-81, although the number of coordinates will grow in basic game extensions, and require 2 bytes for larger-order gameboards such as <a href="http://www.samurai-sudoku.com/sudoku3.gif" rel="nofollow noreferrer">"Samurai" Sudoku</a>.) So the first turn on a given game requires only 2 bytes, the second turn bytes, etc., up to between 50 and 70 turns on an 81 cell gameboard.  Additionally, because it's a square grid, we can reduce for symmetry.  But even with an average number of turns at 50, that's only about 40,000 games for 200MB.  </p>
  
  <ul>
  <li>Weighting Openings</li>
  </ul>
  
  <p>My feeling is that reinforcement would be useful in weighting openings.  If the game data is a string, these strings can be compared, and the smaller the sample size (the fewer the turns included,) the more connections there will be. In this case, the sequence doesn't matter, only the set of value/coordinates for a given turn. Abstraction can be utilized in that certain individual value/coordinates are interchangeable. <strong>My thought is the automata can weight openings based on how often they lead to desirable outcomes in the form of a win.</strong>  </p>
  
  <ul>
  <li>Weighting Heuristics</li>
  </ul>
  
  <p>Since we're having good, initial results with heuristics, and these are the most efficient method of decision-making, <strong>I'm thinking about weighting evaluation functions so that certain heuristics take precedence under different conditions.</strong>  (For instance, when to expand vs. when to consolidate. When to make a choice with immediate benefit over a choice with long-term benefit. Introduction of meta-strategies that modify foundation strategies.)</p>
  
  <ul>
  <li>Database pruning</li>
  </ul>
  
  <p>Because the allocated volume will be capped, <strong>it's probably going to be necessary to "prune" the database when info is no longer relevant. (For instance, when a new strategy emerges that renders previous strategies obsolete.)</strong> We also probably need a method to help the automata to recognize such situations, so it doesn't persist in potentially obsolete strategies for more than two games without starting to try alternatives.  </p>
  
  <hr>
  
  <p><strong>Q: Can reinforcement learning be meaningfully applied toward these goals under these restrictions?</strong>  </p>
  
  <p><strong>Q: Are my inclinations for approaching this useful or problematic?</strong></p>
  
  <p><strong>Q: Are there methods I'm not considering that could be applicable under these restrictions?</strong></p>
  
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2017-10-27T02:32:42.480Z
Comments.Text: I just want to point out that training and executing don't have to be done on the same device. E.g. (as far as I know) Tensorflow allows you to train a NN on a powerful machine (e.g. PC with GPUs) and then deploy the trained NN on a much less powerful machine, e.g. an Android phone.
Comments.UserDisplayName: user12889
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2017-10-27T17:30:28.817Z
Comments.Text: @user12889 That's a great point and you're not the only one to indicate this NN capability.  Part of this initiative will be to see if we can do "mini-NNs" on the local mobile devices, where the local automata get "tuned" by playing against their human, with the intent of leveraging evolutionary game theory (the idea being that each discrete automata is unique do to the "personalization".)  But if we find we need to produce strong AI for an expert-player version of the product, the NN capability you mention will surely be extremely effective!
Comments.UserDisplayName: DukeZhou
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2018-09-16T22:48:14.810Z
Comments.Text: @DuttaA thanks for the poke.  When I originally asked it, I wasn't even sure how valid of a question it was.  *(Luckily, there's been some recent clarification of the approach in terms of the project, where learning algorithms will be the final part, and are still some ways off;)*
Comments.UserDisplayName: DukeZhou
Tags: <reinforcement-learning><game-ai><architecture><combinatorics><hardware-evaluation>
Title: What type of reinforcement learning can I do restricted to ~200MB on an average smartphone?
UserDisplayName: DukeZhou
ThreadText: # What type of reinforcement learning can I do restricted to ~200MB on an average smartphone?
  
  (Asked by: username_0 on 2017-08-07)
  
  This concerns a set of finite, non-trivial, combinatorial games [[M]](http://www.fundamentalcombinatronics.com/rules-of-m/) in the form of an app. [A sample game can be found here](http://www.mclassgames.com/powerofm/).
  
  Because this is a mass market product, we can't take up too much space, and the AI needs to be able to run locally since connectivity cannot be assumed. The current size of the Android kernel is < 7MB.
  
  The goal is not sheer AI strength, but respectable AI strength, sufficient to beat the above-average human player. *(The current strongest, weak automata, using a few heuristics, is already capable of beating the average human player.)*
  
  Because the games are finite, the gametrees eventually become tractable, allowing for perfect endgames. Resource stealing strategies and trap-avoidance can also be effected with shallow look-ahead at all phases, and the patterns are much easier for automata to recognize than for humans.
  
  **In this context, reinforcement would be mostly utilized to "tune the automata to the style of the human player," and produce different automata on different devices, which could subsequently play against each other as proxies for their human partners.**
  
  The game data can be stored efficiently, initially requiring only 2 bytes per position (a value 0-9 and a coordinate 1-81, although the number of coordinates will grow in basic game extensions, and require 2 bytes for larger-order gameboards such as ["Samurai" Sudoku](http://www.samurai-sudoku.com/sudoku3.gif).) So the first turn on a given game requires only 2 bytes, the second turn bytes, etc., up to between 50 and 70 turns on an 81 cell gameboard. Additionally, because it's a square grid, we can reduce for symmetry. But even with an average number of turns at 50, that's only about 40,000 games for 200MB.
  
  * Weighting Openings
  
  My feeling is that reinforcement would be useful in weighting openings. If the game data is a string, these strings can be compared, and the smaller the sample size (the fewer the turns included,) the more connections there will be. In this case, the sequence doesn't matter, only the set of value/coordinates for a given turn. Abstraction can be utilized in that certain individual value/coordinates are interchangeable. **My thought is the automata can weight openings based on how often they lead to desirable outcomes in the form of a win.**
  
  * Weighting Heuristics
  
  Since we're having good, initial results with heuristics, and these are the most efficient method of decision-making, **I'm thinking about weighting evaluation functions so that certain heuristics take precedence under different conditions.** (For instance, when to expand vs. when to consolidate. When to make a choice with immediate benefit over a choice with long-term benefit. Introduction of meta-strategies that modify foundation strategies.)
  
  * Database pruning
  
  Because the allocated volume will be capped, **it's probably going to be necessary to "prune" the database when info is no longer relevant. (For instance, when a new strategy emerges that renders previous strategies obsolete.)** We also probably need a method to help the automata to recognize such situations, so it doesn't persist in potentially obsolete strategies for more than two games without starting to try alternatives.
  
  ---
  
  **Q: Can reinforcement learning be meaningfully applied toward these goals under these restrictions?**
  
  **Q: Are my inclinations for approaching this useful or problematic?**
  
  **Q: Are there methods I'm not considering that could be applicable under these restrictions?**
  
  **Tags:** reinforcement-learning, game-ai, architecture, combinatorics, hardware-evaluation
  
  > **Comments:**
  > **username_1**: I just want to point out that training and executing don't have to be done on the same device. E.g. (as far as I know) Tensorflow allows you to train a NN on a powerful machine (e.g. PC with GPUs) and then deploy the trained NN on a much less powerful machine, e.g. an Android phone.  
  > **username_0**: @username_1 That's a great point and you're not the only one to indicate this NN capability.  Part of this initiative will be to see if we can do "mini-NNs" on the local mobile devices, where the local automata get "tuned" by playing against their human, with the intent of leveraging evolutionary game theory (the idea being that each discrete automata is unique do to the "personalization".)  But if we find we need to produce strong AI for an expert-player version of the product, the NN capability you mention will surely be extremely effective!  
  > **username_0**: @DuttaA thanks for the poke.  When I originally asked it, I wasn't even sure how valid of a question it was.  *(Luckily, there's been some recent clarification of the approach in terms of the project, where learning algorithms will be the final part, and are still some ways off;)*  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2017-08-23. Score: 2
  
  I skimmed through your question and understood that the state/action space is finite, so in this case, RL would be a good option for storage.
  
  The most basic RL technique will keep track of a matrix **Q** &in; ℝ*s*×*a*, where *s* is number of possible states, and *a* is number of possible actions.
  
  In addition to a small overhead of agent's parameters:
  
  * α &in; [0,1] learning parameter
  * γ &in; [0,1] discount rate,
  * ε &in; [0,1] exploration rate
  
  If you have a total of 81 states and 10 actions, then your storage would be in the neighborhood of 850 double words.
  
  If your AI will track different states per location, e.g. possible values 0 to 9 + a null, then your state space will grow to 11 \* 81 = 968 and the final value matrix will be **Q** &in; ℝ968×10 ≈ 20Kb.
  
  ## Update
  
  I am terrible sorry, I made a terrible mistake in my calculations.
  if you are to keep track of 11 possible values in 81 locations, then the correct number should be 1181× double word size, that is a big number.
  
  Of course, if this is Sudoku and you and you are not considering illegal moves as possible states, then definitely this will be a way smaller number, but you will still keep track of an action space that is as big as the state space.
  
  You might need to use function approximation instead, or some sort of action state space compression.
  
  sorry
  
  > **Comments:**
  > **username_0**: Don't apologize!  I really appreciate you taking the time, and this answer is helpful.  Really it's only 9 values over 81 positions, but the game mechanics result with endgame states with ~10 dead cells where no integers can be legally played.  Because of the tractable endgame, we don't need to even go that deep, and merely track openings.  
  > **username_0**: PS I added a slight edit to your first suggestion to the description for the reinforcement-learning tag.  Thanks for getting involved.  If you want to discuss further tweaks, feel free to open a chat, and invite me from a comment here.  
  
  ---
  
  