Answers.Body: <p>Your forward function is not using the previous hidden state.</p>
  
  <p>observe: you pass hidden but never use it.
  </p>
  
  <pre><code>def forward(self, input, hidden):
      layer = self.hiddenWx1(input)
      layer = self.hiddenWx2(layer)
      a_next = self.tanh(layer)
      z = self.z1(a_next)
      z = self.z2(z)
      y_next = self.softmax(z)
      return y_next,a_next
  </code></pre>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-09-06T21:40:52.270Z
Answers.Text: yes correct... sorry for the bad mistake, will modify it.
Answers.UserDisplayName: john7002
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-09-06T21:41:46.757Z
Answers.Text: it happens to the best of us.  I hope this actually solves the issue.
Answers.UserDisplayName: respectful
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-09-06T21:37:37.500Z
Answers.LastActivityDate: 2019-09-06T21:37:37.500Z
Answers.UserDisplayName: respectful
Body: <p>After completing Coursera course from Andrew Ng, I wanted to implement again simple RNN for generating dinosaurs name based on a text file containing around 800 dinosaurs name. 
  This is done with Numpy in coursera, here is a link to a  Jupyter notebook (not my repo) to get strategy and full objective: 
  <a href="https://github.com/Kulbear/deep-learning-coursera/blob/master/Sequence%20Models/Dinosaurus%20Island%20--%20Character%20level%20language%20model%20final%20-%20v3.ipynb" rel="nofollow noreferrer">Here</a></p>
  
  <p>I started similar implementation but in Pytorch, here is the model:</p>
  
  
  
  <pre><code>class RNN(nn.Module):
      def __init__(self,input_size):
          super(RNN, self).__init__()
          print("oo")
          self.hiddenWx1 = nn.Linear(input_size, 100) 
          self.hiddenWx2 = nn.Linear(100, input_size)
          self.z1 = nn.Linear(input_size,100)
          self.z2 = nn.Linear(100,input_size)
          self.tanh = nn.Tanh()
          self.softmax = torch.nn.Softmax(dim=1)
  
      def forward(self, input, hidden):
          layer = self.hiddenWx1(input)
          layer = self.hiddenWx2(layer)
          a_next = self.tanh(layer)
          z = self.z1(a_next)
          z = self.z2(z)
          y_next = self.softmax(z)
          return y_next,a_next
  </code></pre>
  
  <p>Here is the main algorithm of training:</p>
  
  
  
  <pre><code>for word in examples:  # for every dinosaurus name
                  model.zero_grad()
                  hidden= torch.zeros(1, len(ix_to_char)) #initialise hidden to null, ix_to_char is below
                  word_vector = word_tensor(word) # convert each letter of  the current name in one-hot tensors
                  output = torch.zeros(1, len(ix_to_char)) #first input is null
                  loss = 0
                  counter = 0
                  true = torch.LongTensor(len(word)) #will contains the index of each letter.If word is "badu" =&gt; [2,1,4,22,0]
  
                  measured = torch.zeros(len(word)) # will contains the vectors returned by the model for each letter (softmax output) 
  
  
                  for t in range(len(word_vector)): # for each letter of current word
                      true[counter] = char_to_ix[word[counter]] # char_to_ix return the index of letter in dictionary
  
                      output, hidden = model(output, hidden)
  
                      if (counter ==0):
                          measured = output
                      else: #measures is a tensor containing tensors of probability distribution
                          measured = torch.cat((measured,output),dim=0)
                      counter+=1
  
                  loss = nn.CrossEntropyLoss()(measured, true) #
                  loss.backward()
                  optimizer.step()
  </code></pre>
  
  <p>The letter dictionary (ix_to_char) is as follow: </p>
  
  <p>{0: '\n', 1: 'a', 2: 'b', 3: 'c', 4: 'd', 5: 'e', 6: 'f', 7: 'g', 8: 'h', 9: 'i', 10: 'j', 11: 'k', 12: 'l', 13: 'm', 14: 'n', 15: 'o', 16: 'p', 17: 'q', 18: 'r', 19: 's', 20: 't', 21: 'u', 22: 'v', 23: 'w', 24: 'x', 25: 'y', 26: 'z'}</p>
  
  <p>Every 2000 epochs, I sample some new words with this function using torch multimonial to select a letter based on the softmax probability returned by the model:</p>
  
  
  
  <pre><code>def sampling(model):
      idx = -1 
      counter = 0
      newline_character = char_to_ix['\n']
  
      x = torch.zeros(1,len(ix_to_char))
      hidden = torch.zeros(1, len(ix_to_char))
      generated_word=""
  
  
      while (idx != newline_character and counter != 35):
          x,hidden = model(x, hidden)
          #print(x)
          counter+=1
          idx = torch.multinomial(x,1)
          #print(idx.item())
          generated_word+=ix_to_char[idx.item()]
      if counter ==35:
          generated_word+='\n'
      print(generated_word)
  </code></pre>
  
  <p>Here are the results of the first display:</p>
  
  <pre><code>epoch:1, loss:3.256033420562744
  aaasaaauasaaasasauaaaaapsaaaasaaaaa
  
  aaaaaaaaaaaaasaaaoaaaaaauaaaaaaaaaa
  
  taaaauasaasaaaaasaaasaauaaaaaaaausa
  
  uaasaaaaauaaaasasssaauaaaaasaaaaaaa
  
  auaaaaaaaassasaaauaaaaaaaaasasaaaas
  
  epoch:2, loss:3.199960231781006
  aaasaaassussssusssussssssssssusssss
  
  aasaaassssssssssssasusssissssssssss
  
  sasaaassssuosasssssssssssssssssssss
  
  aasassasassusssssssssussssssssssuss
  
  oasaasassssssussssssssussssssssssss
  
  epoch:3, loss:3.263746500015259
  aaaaaaasaaaasaaaaasaaaasaaaaaaaaaaa
  
  aaaaaaasaaaaaaaaaaaaaaaaaaaaaaaaaaa
  
  aaaaaaaaaaaaaaaaaaaaaauaaaaaaaaaaas
  
  aaaaaaaasaaaasraaaaaaaaaaaaaaaaaaaa
  
  aaaaaaaaaaaaauusaaaaauaaaaaaaaaaaaa
  </code></pre>
  
  <p>It doesn't work but I have no idea how to fix the issue. 
  With no training at all, the sampling function seems to work as the returned words seem complete random:</p>
  
  <pre><code>hbtpsbykkxvlah
  
  ttiwlzxdxabzmbdvsapsnwwpaoiasotalft
  </code></pre>
  
  <p>My post may be a bit long, but so far I have no idea what is the issue of my program.</p>
  
  <p>Manny thanks for your help.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-09-05T20:36:11.033Z
Comments.Text: Can you add a link to the dataset. I'll check your model when I get a chance later but I'll need the dataset too.
Comments.UserDisplayName: respectful
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-09-06T08:05:19.040Z
Comments.Text: Here is a link to the dataset (not my repo): https://github.com/carlb15/Dinosaur_Island_Character_Lvl_Language_Model/blob/master/dinos.txt   thanks!
Comments.UserDisplayName: john7002
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-09-06T20:40:24.453Z
Comments.Text: I'm looking into it now.
Comments.UserDisplayName: respectful
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-09-06T21:17:17.073Z
Comments.Text: thanks, just check again and seems that 'output' variable initialized as null vector get a its value updated after calling model the first time, and then this value is never updated..
Comments.UserDisplayName: john7002
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-09-06T21:19:06.013Z
Comments.Text: That would explain all the a's
Comments.UserDisplayName: respectful
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-09-06T21:27:29.530Z
Comments.Text: I just noticed that your forward function does not use the hidden state you pass it
Comments.UserDisplayName: respectful
Tags: <recurrent-neural-networks><pytorch>
Title: Issue at training simple RNN for word generation
UserDisplayName: john7002
ThreadText: # Issue at training simple RNN for word generation
  
  (Asked by: username_0 on 2019-09-05)
  
  After completing Coursera course from Andrew Ng, I wanted to implement again simple RNN for generating dinosaurs name based on a text file containing around 800 dinosaurs name.
  This is done with Numpy in coursera, here is a link to a Jupyter notebook (not my repo) to get strategy and full objective:
  [Here](https://github.com/Kulbear/deep-learning-coursera/blob/master/Sequence%20Models/Dinosaurus%20Island%20--%20Character%20level%20language%20model%20final%20-%20v3.ipynb)
  
  I started similar implementation but in Pytorch, here is the model:
  
  ```
  class RNN(nn.Module):
      def __init__(self,input_size):
          super(RNN, self).__init__()
          print("oo")
          self.hiddenWx1 = nn.Linear(input_size, 100) 
          self.hiddenWx2 = nn.Linear(100, input_size)
          self.z1 = nn.Linear(input_size,100)
          self.z2 = nn.Linear(100,input_size)
          self.tanh = nn.Tanh()
          self.softmax = torch.nn.Softmax(dim=1)
  
      def forward(self, input, hidden):
          layer = self.hiddenWx1(input)
          layer = self.hiddenWx2(layer)
          a_next = self.tanh(layer)
          z = self.z1(a_next)
          z = self.z2(z)
          y_next = self.softmax(z)
          return y_next,a_next
  
  ```
  
  Here is the main algorithm of training:
  
  ```
  for word in examples:  # for every dinosaurus name
                  model.zero_grad()
                  hidden= torch.zeros(1, len(ix_to_char)) #initialise hidden to null, ix_to_char is below
                  word_vector = word_tensor(word) # convert each letter of  the current name in one-hot tensors
                  output = torch.zeros(1, len(ix_to_char)) #first input is null
                  loss = 0
                  counter = 0
                  true = torch.LongTensor(len(word)) #will contains the index of each letter.If word is "badu" => [2,1,4,22,0]
  
                  measured = torch.zeros(len(word)) # will contains the vectors returned by the model for each letter (softmax output) 
  
                  for t in range(len(word_vector)): # for each letter of current word
                      true[counter] = char_to_ix[word[counter]] # char_to_ix return the index of letter in dictionary
  
                      output, hidden = model(output, hidden)
  
                      if (counter ==0):
                          measured = output
                      else: #measures is a tensor containing tensors of probability distribution
                          measured = torch.cat((measured,output),dim=0)
                      counter+=1
  
                  loss = nn.CrossEntropyLoss()(measured, true) #
                  loss.backward()
                  optimizer.step()
  
  ```
  
  The letter dictionary (ix\_to\_char) is as follow:
  
  {0: '\n', 1: 'a', 2: 'b', 3: 'c', 4: 'd', 5: 'e', 6: 'f', 7: 'g', 8: 'h', 9: 'i', 10: 'j', 11: 'k', 12: 'l', 13: 'm', 14: 'n', 15: 'o', 16: 'p', 17: 'q', 18: 'r', 19: 's', 20: 't', 21: 'u', 22: 'v', 23: 'w', 24: 'x', 25: 'y', 26: 'z'}
  
  Every 2000 epochs, I sample some new words with this function using torch multimonial to select a letter based on the softmax probability returned by the model:
  
  ```
  def sampling(model):
      idx = -1 
      counter = 0
      newline_character = char_to_ix['\n']
  
      x = torch.zeros(1,len(ix_to_char))
      hidden = torch.zeros(1, len(ix_to_char))
      generated_word=""
  
      while (idx != newline_character and counter != 35):
          x,hidden = model(x, hidden)
          #print(x)
          counter+=1
          idx = torch.multinomial(x,1)
          #print(idx.item())
          generated_word+=ix_to_char[idx.item()]
      if counter ==35:
          generated_word+='\n'
      print(generated_word)
  
  ```
  
  Here are the results of the first display:
  
  ```
  epoch:1, loss:3.256033420562744
  aaasaaauasaaasasauaaaaapsaaaasaaaaa
  
  aaaaaaaaaaaaasaaaoaaaaaauaaaaaaaaaa
  
  taaaauasaasaaaaasaaasaauaaaaaaaausa
  
  uaasaaaaauaaaasasssaauaaaaasaaaaaaa
  
  auaaaaaaaassasaaauaaaaaaaaasasaaaas
  
  epoch:2, loss:3.199960231781006
  aaasaaassussssusssussssssssssusssss
  
  aasaaassssssssssssasusssissssssssss
  
  sasaaassssuosasssssssssssssssssssss
  
  aasassasassusssssssssussssssssssuss
  
  oasaasassssssussssssssussssssssssss
  
  epoch:3, loss:3.263746500015259
  aaaaaaasaaaasaaaaasaaaasaaaaaaaaaaa
  
  aaaaaaasaaaaaaaaaaaaaaaaaaaaaaaaaaa
  
  aaaaaaaaaaaaaaaaaaaaaauaaaaaaaaaaas
  
  aaaaaaaasaaaasraaaaaaaaaaaaaaaaaaaa
  
  aaaaaaaaaaaaauusaaaaauaaaaaaaaaaaaa
  
  ```
  
  It doesn't work but I have no idea how to fix the issue.
  With no training at all, the sampling function seems to work as the returned words seem complete random:
  
  ```
  hbtpsbykkxvlah
  
  ttiwlzxdxabzmbdvsapsnwwpaoiasotalft
  
  ```
  
  My post may be a bit long, but so far I have no idea what is the issue of my program.
  
  Manny thanks for your help.
  
  **Tags:** recurrent-neural-networks, pytorch
  
  > **Comments:**
  > **username_1**: Can you add a link to the dataset. I'll check your model when I get a chance later but I'll need the dataset too.  
  > **username_0**: Here is a link to the dataset (not my repo): https://github.com/carlb15/Dinosaur_Island_Character_Lvl_Language_Model/blob/master/dinos.txt   thanks!  
  > **username_1**: I'm looking into it now.  
  > **username_0**: thanks, just check again and seems that 'output' variable initialized as null vector get a its value updated after calling model the first time, and then this value is never updated..  
  > **username_1**: That would explain all the a's  
  > **username_1**: I just noticed that your forward function does not use the hidden state you pass it  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2019-09-06. Score: 1
  
  Your forward function is not using the previous hidden state.
  
  observe: you pass hidden but never use it.
  
  ```
  def forward(self, input, hidden):
      layer = self.hiddenWx1(input)
      layer = self.hiddenWx2(layer)
      a_next = self.tanh(layer)
      z = self.z1(a_next)
      z = self.z2(z)
      y_next = self.softmax(z)
      return y_next,a_next
  
  ```
  
  > **Comments:**
  > **username_0**: yes correct... sorry for the bad mistake, will modify it.  
  > **username_1**: it happens to the best of us.  I hope this actually solves the issue.  
  
  ---
  
  