<p>Your forward function is not using the previous hidden state.</p>

<p>observe: you pass hidden but never use it.
</p>

<pre><code>def forward(self, input, hidden):
    layer = self.hiddenWx1(input)
    layer = self.hiddenWx2(layer)
    a_next = self.tanh(layer)
    z = self.z1(a_next)
    z = self.z2(z)
    y_next = self.softmax(z)
    return y_next,a_next
</code></pre>

CC BY-SA 4.0
2019-09-06T21:40:52.270Z
yes correct... sorry for the bad mistake, will modify it.
john7002
CC BY-SA 4.0
2019-09-06T21:41:46.757Z
it happens to the best of us.  I hope this actually solves the issue.
respectful
CC BY-SA 4.0
2019-09-06T21:37:37.500Z
2019-09-06T21:37:37.500Z
respectful
<p>After completing Coursera course from Andrew Ng, I wanted to implement again simple RNN for generating dinosaurs name based on a text file containing around 800 dinosaurs name. 
This is done with Numpy in coursera, here is a link to a  Jupyter notebook (not my repo) to get strategy and full objective: 
<a href="https://github.com/Kulbear/deep-learning-coursera/blob/master/Sequence%20Models/Dinosaurus%20Island%20--%20Character%20level%20language%20model%20final%20-%20v3.ipynb" rel="nofollow noreferrer">Here</a></p>

<p>I started similar implementation but in Pytorch, here is the model:</p>



<pre><code>class RNN(nn.Module):
    def __init__(self,input_size):
        super(RNN, self).__init__()
        print("oo")
        self.hiddenWx1 = nn.Linear(input_size, 100) 
        self.hiddenWx2 = nn.Linear(100, input_size)
        self.z1 = nn.Linear(input_size,100)
        self.z2 = nn.Linear(100,input_size)
        self.tanh = nn.Tanh()
        self.softmax = torch.nn.Softmax(dim=1)

    def forward(self, input, hidden):
        layer = self.hiddenWx1(input)
        layer = self.hiddenWx2(layer)
        a_next = self.tanh(layer)
        z = self.z1(a_next)
        z = self.z2(z)
        y_next = self.softmax(z)
        return y_next,a_next
</code></pre>

<p>Here is the main algorithm of training:</p>



<pre><code>for word in examples:  # for every dinosaurus name
                model.zero_grad()
                hidden= torch.zeros(1, len(ix_to_char)) #initialise hidden to null, ix_to_char is below
                word_vector = word_tensor(word) # convert each letter of  the current name in one-hot tensors
                output = torch.zeros(1, len(ix_to_char)) #first input is null
                loss = 0
                counter = 0
                true = torch.LongTensor(len(word)) #will contains the index of each letter.If word is "badu" =&gt; [2,1,4,22,0]

                measured = torch.zeros(len(word)) # will contains the vectors returned by the model for each letter (softmax output) 


                for t in range(len(word_vector)): # for each letter of current word
                    true[counter] = char_to_ix[word[counter]] # char_to_ix return the index of letter in dictionary

                    output, hidden = model(output, hidden)

                    if (counter ==0):
                        measured = output
                    else: #measures is a tensor containing tensors of probability distribution
                        measured = torch.cat((measured,output),dim=0)
                    counter+=1

                loss = nn.CrossEntropyLoss()(measured, true) #
                loss.backward()
                optimizer.step()
</code></pre>

<p>The letter dictionary (ix_to_char) is as follow: </p>

<p>{0: '\n', 1: 'a', 2: 'b', 3: 'c', 4: 'd', 5: 'e', 6: 'f', 7: 'g', 8: 'h', 9: 'i', 10: 'j', 11: 'k', 12: 'l', 13: 'm', 14: 'n', 15: 'o', 16: 'p', 17: 'q', 18: 'r', 19: 's', 20: 't', 21: 'u', 22: 'v', 23: 'w', 24: 'x', 25: 'y', 26: 'z'}</p>

<p>Every 2000 epochs, I sample some new words with this function using torch multimonial to select a letter based on the softmax probability returned by the model:</p>



<pre><code>def sampling(model):
    idx = -1 
    counter = 0
    newline_character = char_to_ix['\n']

    x = torch.zeros(1,len(ix_to_char))
    hidden = torch.zeros(1, len(ix_to_char))
    generated_word=""


    while (idx != newline_character and counter != 35):
        x,hidden = model(x, hidden)
        #print(x)
        counter+=1
        idx = torch.multinomial(x,1)
        #print(idx.item())
        generated_word+=ix_to_char[idx.item()]
    if counter ==35:
        generated_word+='\n'
    print(generated_word)
</code></pre>

<p>Here are the results of the first display:</p>

<pre><code>epoch:1, loss:3.256033420562744
aaasaaauasaaasasauaaaaapsaaaasaaaaa

aaaaaaaaaaaaasaaaoaaaaaauaaaaaaaaaa

taaaauasaasaaaaasaaasaauaaaaaaaausa

uaasaaaaauaaaasasssaauaaaaasaaaaaaa

auaaaaaaaassasaaauaaaaaaaaasasaaaas

epoch:2, loss:3.199960231781006
aaasaaassussssusssussssssssssusssss

aasaaassssssssssssasusssissssssssss

sasaaassssuosasssssssssssssssssssss

aasassasassusssssssssussssssssssuss

oasaasassssssussssssssussssssssssss

epoch:3, loss:3.263746500015259
aaaaaaasaaaasaaaaasaaaasaaaaaaaaaaa

aaaaaaasaaaaaaaaaaaaaaaaaaaaaaaaaaa

aaaaaaaaaaaaaaaaaaaaaauaaaaaaaaaaas

aaaaaaaasaaaasraaaaaaaaaaaaaaaaaaaa

aaaaaaaaaaaaauusaaaaauaaaaaaaaaaaaa
</code></pre>

<p>It doesn't work but I have no idea how to fix the issue. 
With no training at all, the sampling function seems to work as the returned words seem complete random:</p>

<pre><code>hbtpsbykkxvlah

ttiwlzxdxabzmbdvsapsnwwpaoiasotalft
</code></pre>

<p>My post may be a bit long, but so far I have no idea what is the issue of my program.</p>

<p>Manny thanks for your help.</p>

CC BY-SA 4.0
2019-09-05T20:36:11.033Z
Can you add a link to the dataset. I'll check your model when I get a chance later but I'll need the dataset too.
respectful
CC BY-SA 4.0
2019-09-06T08:05:19.040Z
Here is a link to the dataset (not my repo): https://github.com/carlb15/Dinosaur_Island_Character_Lvl_Language_Model/blob/master/dinos.txt   thanks!
john7002
CC BY-SA 4.0
2019-09-06T20:40:24.453Z
I'm looking into it now.
respectful
CC BY-SA 4.0
2019-09-06T21:17:17.073Z
thanks, just check again and seems that 'output' variable initialized as null vector get a its value updated after calling model the first time, and then this value is never updated..
john7002
CC BY-SA 4.0
2019-09-06T21:19:06.013Z
That would explain all the a's
respectful
CC BY-SA 4.0
2019-09-06T21:27:29.530Z
I just noticed that your forward function does not use the hidden state you pass it
respectful
<recurrent-neural-networks><pytorch>
Issue at training simple RNN for word generation
john7002
# Issue at training simple RNN for word generation

(Asked by: username_0 on 2019-09-05)

After completing Coursera course from Andrew Ng, I wanted to implement again simple RNN for generating dinosaurs name based on a text file containing around 800 dinosaurs name. 
This is done with Numpy in coursera, here is a link to a  Jupyter notebook (not my repo) to get strategy and full objective: 
Here
I started similar implementation but in Pytorch, here is the model:
class RNN(nn.Module):
    def __init__(self,input_size):
        super(RNN, self).__init__()
        print("oo")
        self.hiddenWx1 = nn.Linear(input_size, 100) 
        self.hiddenWx2 = nn.Linear(100, input_size)
        self.z1 = nn.Linear(input_size,100)
        self.z2 = nn.Linear(100,input_size)
        self.tanh = nn.Tanh()
        self.softmax = torch.nn.Softmax(dim=1)

    def forward(self, input, hidden):
        layer = self.hiddenWx1(input)
        layer = self.hiddenWx2(layer)
        a_next = self.tanh(layer)
        z = self.z1(a_next)
        z = self.z2(z)
        y_next = self.softmax(z)
        return y_next,a_next

Here is the main algorithm of training:
for word in examples:  # for every dinosaurus name
                model.zero_grad()
                hidden= torch.zeros(1, len(ix_to_char)) #initialise hidden to null, ix_to_char is below
                word_vector = word_tensor(word) # convert each letter of  the current name in one-hot tensors
                output = torch.zeros(1, len(ix_to_char)) #first input is null
                loss = 0
                counter = 0
                true = torch.LongTensor(len(word)) #will contains the index of each letter.If word is "badu" => [2,1,4,22,0]

                measured = torch.zeros(len(word)) # will contains the vectors returned by the model for each letter (softmax output) 


                for t in range(len(word_vector)): # for each letter of current word
                    true[counter] = char_to_ix[word[counter]] # char_to_ix return the index of letter in dictionary

                    output, hidden = model(output, hidden)

                    if (counter ==0):
                        measured = output
                    else: #measures is a tensor containing tensors of probability distribution
                        measured = torch.cat((measured,output),dim=0)
                    counter+=1

                loss = nn.CrossEntropyLoss()(measured, true) #
                loss.backward()
                optimizer.step()

The letter dictionary (ix_to_char) is as follow: 
{0: '\n', 1: 'a', 2: 'b', 3: 'c', 4: 'd', 5: 'e', 6: 'f', 7: 'g', 8: 'h', 9: 'i', 10: 'j', 11: 'k', 12: 'l', 13: 'm', 14: 'n', 15: 'o', 16: 'p', 17: 'q', 18: 'r', 19: 's', 20: 't', 21: 'u', 22: 'v', 23: 'w', 24: 'x', 25: 'y', 26: 'z'}
Every 2000 epochs, I sample some new words with this function using torch multimonial to select a letter based on the softmax probability returned by the model:
def sampling(model):
    idx = -1 
    counter = 0
    newline_character = char_to_ix['\n']

    x = torch.zeros(1,len(ix_to_char))
    hidden = torch.zeros(1, len(ix_to_char))
    generated_word=""


    while (idx != newline_character and counter != 35):
        x,hidden = model(x, hidden)
        #print(x)
        counter+=1
        idx = torch.multinomial(x,1)
        #print(idx.item())
        generated_word+=ix_to_char[idx.item()]
    if counter ==35:
        generated_word+='\n'
    print(generated_word)

Here are the results of the first display:
epoch:1, loss:3.256033420562744
aaasaaauasaaasasauaaaaapsaaaasaaaaa

aaaaaaaaaaaaasaaaoaaaaaauaaaaaaaaaa

taaaauasaasaaaaasaaasaauaaaaaaaausa

uaasaaaaauaaaasasssaauaaaaasaaaaaaa

auaaaaaaaassasaaauaaaaaaaaasasaaaas

epoch:2, loss:3.199960231781006
aaasaaassussssusssussssssssssusssss

aasaaassssssssssssasusssissssssssss

sasaaassssuosasssssssssssssssssssss

aasassasassusssssssssussssssssssuss

oasaasassssssussssssssussssssssssss

epoch:3, loss:3.263746500015259
aaaaaaasaaaasaaaaasaaaasaaaaaaaaaaa

aaaaaaasaaaaaaaaaaaaaaaaaaaaaaaaaaa

aaaaaaaaaaaaaaaaaaaaaauaaaaaaaaaaas

aaaaaaaasaaaasraaaaaaaaaaaaaaaaaaaa

aaaaaaaaaaaaauusaaaaauaaaaaaaaaaaaa

It doesn't work but I have no idea how to fix the issue. 
With no training at all, the sampling function seems to work as the returned words seem complete random:
hbtpsbykkxvlah

ttiwlzxdxabzmbdvsapsnwwpaoiasotalft

My post may be a bit long, but so far I have no idea what is the issue of my program.
Manny thanks for your help.

**Tags:** recurrent-neural-networks, pytorch

> **Comments:**
> **username_1**: Can you add a link to the dataset. I'll check your model when I get a chance later but I'll need the dataset too.  
> **username_0**: Here is a link to the dataset (not my repo): https://github.com/carlb15/Dinosaur_Island_Character_Lvl_Language_Model/blob/master/dinos.txt   thanks!  
> **username_1**: I'm looking into it now.  
> **username_0**: thanks, just check again and seems that 'output' variable initialized as null vector get a its value updated after calling model the first time, and then this value is never updated..  
> **username_1**: That would explain all the a's  
> **username_1**: I just noticed that your forward function does not use the hidden state you pass it  

---

## 1 answers

### Answer by username_1 on 2019-09-06. Score: 1

Your forward function is not using the previous hidden state.
observe: you pass hidden but never use it.

def forward(self, input, hidden):
    layer = self.hiddenWx1(input)
    layer = self.hiddenWx2(layer)
    a_next = self.tanh(layer)
    z = self.z1(a_next)
    z = self.z2(z)
    y_next = self.softmax(z)
    return y_next,a_next

> **Comments:**
> **username_0**: yes correct... sorry for the bad mistake, will modify it.  
> **username_1**: it happens to the best of us.  I hope this actually solves the issue.  

---

