Answers.Body: <p>A quite effective way to solve Wordle is to maintain an active list of all five-letter words that meet criteria so far, choose one of them completely at random, then update the list of possible words based on new knowledge from the response.</p>
  <p>A slightly smarter way would be to pick words from the subset of remaining ones that should maximise information gathered from the response on average. I believe I read a headline recently that &quot;crane&quot; is a good starting English word for that reason.</p>
  <p>The trouble for you here is that those approaches do not involve ML in any form. Relatively simple logic and analysis are already pretty good at playing Wordle automatically.</p>
  <p>However, to stick with the question as asked, your problem is how to get a machine learning algorithm to choose between maybe 10,000 options.</p>
  <p>You have a couple of choices:</p>
  <ul>
  <li><p>Brute force. Number the options, have the algorithm choose directly. If you do this, I recommend that you also filter out logically impossible words, as that makes the problem far easier. The 10,000 options may seem large at first, but actually this should be within reach of a personal/hobby setup. It will work mainly because it is possible to simulate the results of games very fast. So generating the training data within RL will be fast, the agent can get to see millions of games in a reasonable time frame, training with each target word thousands of times on average.</p>
  </li>
  <li><p>Actions represent additional constraints on word choice. For a simple example, you could have the agent decide whether the first unknown letter it wants to try is a vowel or consonant = just two actions. You can use that to further restrict word choice from the remaining words. This influences an otherwise random word selector, and the agent should learn situations in which to pick words with certain traits.</p>
  </li>
  <li><p>Actions describe a &quot;word vector&quot;, and the selector picks the closest valid word to that vector, breaking ties randomly. In order to do this, you need to generate some short vector of each word in the dictionary. For a simple example* you might choose <code>[part_of_speech, popularity]</code> to qualify each word, scoring <code>part_of_speech</code> as 1 for noun, -1 for verb and 0 for everything else and <code>popularity</code> as 1 for common word, -1 for rare word, and 0 for moderately frequent. Then your action output would be a choice from the 9 combinations possible in this vector.</p>
  </li>
  </ul>
  <p>None of these approaches are simple, so not great if you are just starting in AI/ML, and I would recommend you tackle a simpler toy problem or two using RL before trying to write your Wordle training routine. Going straight to Wordle without solving smaller problems first is likely to cause a lot of incorrect decisions on your part that you won't spot and will frustrate you.</p>
  <hr />
  <p>* I have given a very simple example, but this could logically lead to learned word embeddings, which are an important topic in natural language processing. These are typically quite large real-valued vectors though (maybe 64 or 128 elements) which would be an even larger space to learn about than the fixed list of words. So full word embeddings like word2vec and Glove are not really suitable for the Wordle puzzle.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-02-12T13:22:36.340Z
Answers.Text: Thanks for the suggestions! I'm not sure what you mean by "logically impossible words" though.
Answers.UserDisplayName: Yuval Amir
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-02-12T13:32:56.347Z
Answers.Text: @YuvalAmir: After you choose your first word "hello" you may know that the word contains an "e"  (but not second position) and the "l" in fourth position is correct. That also implies "h" and "o" are not in the target word. So you can immediately remove all words from consideration that contain "h" or "o", that have "e" in second position, that have no "e" at all (because there is at least one), and that don't have "l" in fourth position. So you can keep "cable" and "eagle" in the list, but not "helps" because the latter is inconsistent with the new feedback.
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-02-12T13:35:12.220Z
Answers.Text: oh I get it. Now that I think about it, could the idea of the "word vector" be used by having the ai describe what letters it wants and you find the closest fitting word?
Answers.UserDisplayName: Yuval Amir
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-02-12T13:36:41.270Z
Answers.Text: @YuvalAmir Yes it could, although overall that is an even larger space than 10,000 words, it may be simpler to code and be reasonably effective.
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-02-13T16:45:10.817Z
Answers.Text: "pick words from the subset of remaining ones": the optimal strategy has not this constraint, just to recover as much information as possible (that is, test a word that splits the set of valid words in equal sized sets).
Answers.UserDisplayName: pasaba por aqui
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-02-13T16:57:28.883Z
Answers.Text: @pasabaporaqui Depends which ones you count. I looked this up, and found this - https://wordfinderx.com/5-letter-words which varies considerably on what it counts, so I went for 10,000 as a "typical" number for choices. The actual list used by Wordle is only ~2500 words - see e.g. https://gist.github.com/cfreshman/a03ef2cba789d8cf00c08f767e0fad7b#file-wordle-answers-alphabetical-txt-L2315
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-02-13T17:28:44.203Z
Answers.Text: "word2vec" maps distance between words taken into account their meanings (inferred from large corpus). Wordle distance between words is purely about word spell. Use word2vec or similar in this case will be a bad choice.
Answers.UserDisplayName: pasaba por aqui
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-02-13T17:31:00.253Z
Answers.Text: @pasabaporaqui Yes, I also explain why they are also not a great choice due to large space. However, they are an interesting way to characterise words in ML, as an aside to the idea of using word characteristic choice as an action space.
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-02-13T17:31:57.740Z
Answers.Text: Finally, all the explanations about "actions" is also unrelated with the main strategy: use as next word a word that, in average, will discard the greatest amount of words still valid.
Answers.UserDisplayName: pasaba por aqui
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-02-13T17:37:08.230Z
Answers.Text: @pasabaporaqui I am not sure what you are getting at? OP wants an answer about RL, not about best strategy for Wordle. I explain the difference at start of the answer. RL works with terminology of actions, and if OP wants to not have a large action space, alternatives are suggested that should work, and might be trainable. The alternatives presented in the answer are not expected to create an optimal Wordle solver, but a *trainable* one that uses ML
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-02-12T13:04:47.923Z
Answers.LastActivityDate: 2022-02-12T13:10:32.120Z
Answers.LastEditDate: 2022-02-12T13:10:32.120Z
Answers.UserDisplayName: Neil Slater
Body: <p>I'm trying to train an AI using reinforcement learning to play the game Wordle.</p>
  <p>The way the game works is that there is a secret five letter code word that you need to guess, and every time you guess a word (it has to be an actual word it can't just be any five letters) the game tells you what letters of your guess are correct and what letters of your guess are in the code word but not in the correct position. If you guess the word in six guesses or less, you win.</p>
  <p>I want every step the AI to choose a string from a list of valid guesses. Afterwards I will reward it for discovering letters for the first time and discovering their position for the first time. It will be penalized for losing and rewarded for winning based on how early it won.</p>
  <p>The problem is that I can't just number each word and have it choose a number (I think?) because the actual content of the string it chooses matter.</p>
  <p>How should I tackle this issue?</p>
  <p>One thing that might be worth mentioning is that while I'm aware this could be solved algorithmically without the use of reinforcement learning, I want to try this approach anyway. This is less about the finale product and more of an excuse to get familiar with machine learning.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-02-11T21:56:26.363Z
Comments.Text: re: "This is my first machine learning project", you can most likely write a strong Wordle bot without using ML. The ML component is likely to be complex due to the action space, as you've noticed. If learning ML/RL is the goal, you will be a lot better off picking a smaller challenge.
Comments.UserDisplayName: Neil Slater
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-02-12T08:44:53.953Z
Comments.Text: For people not familiar with this game, you need to explain how the "content of the string matters". Moreover, there's not description of how you're using RL here.
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-02-12T10:31:27.993Z
Comments.Text: @nbro thanks for the feedback. I added those to the post.
Comments.UserDisplayName: Yuval Amir
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-02-12T12:26:39.760Z
Comments.Text: I have heard of, but not played Wordle. This fact is important to possible strategies when designing an ML component: Could you explain whether Wordle only accepts valid 5-letter words as input, or if it will accept and score non-words like "ABCDE"?
Comments.UserDisplayName: Neil Slater
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-02-16T02:24:55.307Z
Comments.Text: What do you mean by "The problem is that I can't just number each word and have it choose a number (I think?) because the actual content of the string it chooses matter" ?
Comments.UserDisplayName: Nike Dattani
Tags: <reinforcement-learning><tensorflow>
Title: How to encode actions for training a Wordle agent?
UserDisplayName: Yuval Amir
ThreadText: # How to encode actions for training a Wordle agent?
  
  (Asked by: username_0 on 2022-02-11)
  
  I'm trying to train an AI using reinforcement learning to play the game Wordle.
  
  The way the game works is that there is a secret five letter code word that you need to guess, and every time you guess a word (it has to be an actual word it can't just be any five letters) the game tells you what letters of your guess are correct and what letters of your guess are in the code word but not in the correct position. If you guess the word in six guesses or less, you win.
  
  I want every step the AI to choose a string from a list of valid guesses. Afterwards I will reward it for discovering letters for the first time and discovering their position for the first time. It will be penalized for losing and rewarded for winning based on how early it won.
  
  The problem is that I can't just number each word and have it choose a number (I think?) because the actual content of the string it chooses matter.
  
  How should I tackle this issue?
  
  One thing that might be worth mentioning is that while I'm aware this could be solved algorithmically without the use of reinforcement learning, I want to try this approach anyway. This is less about the finale product and more of an excuse to get familiar with machine learning.
  
  **Tags:** reinforcement-learning, tensorflow
  
  > **Comments:**
  > **username_1**: re: "This is my first machine learning project", you can most likely write a strong Wordle bot without using ML. The ML component is likely to be complex due to the action space, as you've noticed. If learning ML/RL is the goal, you will be a lot better off picking a smaller challenge.  
  > **username_2**: For people not familiar with this game, you need to explain how the "content of the string matters". Moreover, there's not description of how you're using RL here.  
  > **username_0**: @username_2 thanks for the feedback. I added those to the post.  
  > **username_1**: I have heard of, but not played Wordle. This fact is important to possible strategies when designing an ML component: Could you explain whether Wordle only accepts valid 5-letter words as input, or if it will accept and score non-words like "ABCDE"?  
  > **username_3**: What do you mean by "The problem is that I can't just number each word and have it choose a number (I think?) because the actual content of the string it chooses matter" ?  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2022-02-12. Score: 2 | Accepted answer
  
  A quite effective way to solve Wordle is to maintain an active list of all five-letter words that meet criteria so far, choose one of them completely at random, then update the list of possible words based on new knowledge from the response.
  
  A slightly smarter way would be to pick words from the subset of remaining ones that should maximise information gathered from the response on average. I believe I read a headline recently that "crane" is a good starting English word for that reason.
  
  The trouble for you here is that those approaches do not involve ML in any form. Relatively simple logic and analysis are already pretty good at playing Wordle automatically.
  
  However, to stick with the question as asked, your problem is how to get a machine learning algorithm to choose between maybe 10,000 options.
  
  You have a couple of choices:
  
  * Brute force. Number the options, have the algorithm choose directly. If you do this, I recommend that you also filter out logically impossible words, as that makes the problem far easier. The 10,000 options may seem large at first, but actually this should be within reach of a personal/hobby setup. It will work mainly because it is possible to simulate the results of games very fast. So generating the training data within RL will be fast, the agent can get to see millions of games in a reasonable time frame, training with each target word thousands of times on average.
  * Actions represent additional constraints on word choice. For a simple example, you could have the agent decide whether the first unknown letter it wants to try is a vowel or consonant = just two actions. You can use that to further restrict word choice from the remaining words. This influences an otherwise random word selector, and the agent should learn situations in which to pick words with certain traits.
  * Actions describe a "word vector", and the selector picks the closest valid word to that vector, breaking ties randomly. In order to do this, you need to generate some short vector of each word in the dictionary. For a simple example\* you might choose `[part_of_speech, popularity]` to qualify each word, scoring `part_of_speech` as 1 for noun, -1 for verb and 0 for everything else and `popularity` as 1 for common word, -1 for rare word, and 0 for moderately frequent. Then your action output would be a choice from the 9 combinations possible in this vector.
  
  None of these approaches are simple, so not great if you are just starting in AI/ML, and I would recommend you tackle a simpler toy problem or two using RL before trying to write your Wordle training routine. Going straight to Wordle without solving smaller problems first is likely to cause a lot of incorrect decisions on your part that you won't spot and will frustrate you.
  
  ---
  
  \* I have given a very simple example, but this could logically lead to learned word embeddings, which are an important topic in natural language processing. These are typically quite large real-valued vectors though (maybe 64 or 128 elements) which would be an even larger space to learn about than the fixed list of words. So full word embeddings like word2vec and Glove are not really suitable for the Wordle puzzle.
  
  > **Comments:**
  > **username_0**: Thanks for the suggestions! I'm not sure what you mean by "logically impossible words" though.  
  > **username_1**: @username_0: After you choose your first word "hello" you may know that the word contains an "e"  (but not second position) and the "l" in fourth position is correct. That also implies "h" and "o" are not in the target word. So you can immediately remove all words from consideration that contain "h" or "o", that have "e" in second position, that have no "e" at all (because there is at least one), and that don't have "l" in fourth position. So you can keep "cable" and "eagle" in the list, but not "helps" because the latter is inconsistent with the new feedback.  
  > **username_0**: oh I get it. Now that I think about it, could the idea of the "word vector" be used by having the ai describe what letters it wants and you find the closest fitting word?  
  > **username_1**: @username_0 Yes it could, although overall that is an even larger space than 10,000 words, it may be simpler to code and be reasonably effective.  
  > **username_4**: "pick words from the subset of remaining ones": the optimal strategy has not this constraint, just to recover as much information as possible (that is, test a word that splits the set of valid words in equal sized sets).  
  > **username_1**: @username_4 Depends which ones you count. I looked this up, and found this - https://wordfinderx.com/5-letter-words which varies considerably on what it counts, so I went for 10,000 as a "typical" number for choices. The actual list used by Wordle is only ~2500 words - see e.g. https://gist.github.com/cfreshman/a03ef2cba789d8cf00c08f767e0fad7b#file-wordle-answers-alphabetical-txt-L2315  
  > **username_4**: "word2vec" maps distance between words taken into account their meanings (inferred from large corpus). Wordle distance between words is purely about word spell. Use word2vec or similar in this case will be a bad choice.  
  > **username_1**: @username_4 Yes, I also explain why they are also not a great choice due to large space. However, they are an interesting way to characterise words in ML, as an aside to the idea of using word characteristic choice as an action space.  
  > **username_4**: Finally, all the explanations about "actions" is also unrelated with the main strategy: use as next word a word that, in average, will discard the greatest amount of words still valid.  
  > **username_1**: @username_4 I am not sure what you are getting at? OP wants an answer about RL, not about best strategy for Wordle. I explain the difference at start of the answer. RL works with terminology of actions, and if OP wants to not have a large action space, alternatives are suggested that should work, and might be trainable. The alternatives presented in the answer are not expected to create an optimal Wordle solver, but a *trainable* one that uses ML  
  
  ---
  
  