RNN for sequence prediction
Hello, Previously I used keras for CNN and so I am a newbie on both PyTorch and RNN. In keras you can write a script for an RNN for sequence prediction like, <SCODE>in_out_neurons = 1
hidden_neurons = 300

model = Sequential()  
model.add(LSTM(hidden_neurons, batch_input_shape=(None, length_of_sequences, in_out_neurons), return_sequences=False))  
model.add(Dense(in_out_neurons))  
model.add(Activation("linear"))  
<ECODE> but when it comes to PyTorch I don’t know how to implement it. I directly translate the code above into below, but it doesn’t work. <SCODE>class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.rnn1 = nn.GRU(input_size=seq_len,
                            hidden_size=128,
                            num_layers=1)
        self.dense1 = nn.Linear(128, 1)

    def forward(self, x, hidden):
        x, hidden = self.rnn1(x, hidden)
        x = self.dense1(x)
        return x, hidden

    def init_hidden(self, batch_size):
        weight = next(self.parameters()).data
        return Variable(weight.new(128, batch_size, 1).zero_())
<ECODE> how can I implement something like the keras code? thank you.
moskomule
Apart from this, your module looks good to me!
apaszke
and For example, if you input a sequence <SCODE>[[[ 0.1,  0.2]],
 [[ 0.1,  0.2]],
 [[ 0.3,  0.1]]]
<ECODE>
moskomule
apaszke
Thanks a lot for your help, finally the code below works, <SCODE>import torch
import torch.nn as nn
from torch.autograd import Variable

features = 1
seq_len = 10
hidden_size = 128
batch_size = 32

class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.rnn1 = nn.GRU(input_size=features,
                            hidden_size=hidden_size,
                            num_layers=1)
        self.dense1 = nn.Linear(hidden_size, 1)

    def forward(self, x, hidden):
        x, hidden = self.rnn1(x, hidden)
        x = x.select(1, seq_len-1).contiguous()
        x = x.view(-1, hidden_size)
        x = self.dense1(x)
        return x, hidden

    def init_hidden(self):
        weight = next(self.parameters()).data
        return Variable(weight.new(1, batch_size, hidden_size).zero_())

model = Net()
model.cuda()
hidden = model.init_hidden()

X_train_1 = X_train[0:batch_size].reshape(seq_len,batch_size,features)
y_train_1 = y_train[0:batch_size]
model.zero_grad()
T = torch.Tensor
X_train_1, y_train_1 = T(X_train_1), T(y_train_1)
X_train_1, y_train_1 = Variable(X_train_1).cuda(), Variable(y_train_1).cuda()

output, hidden = model(X_train_1, Variable(hidden.data))
<ECODE>
moskomule
Thanks for your helping, like I wrote above the script works, “literally” but the loss doesn’t decrease over the epochs, so give me some advice. I think the related parts are, <SCODE>class Net(nn.Module):
    def __init__(self, features, cls_size):
        super(Net, self).__init__()
        self.rnn1 = nn.GRU(input_size=features,
                            hidden_size=hidden_size,
                            num_layers=1)
        self.dense1 = nn.Linear(hidden_size, cls_size)

    def forward(self, x, hidden):
        x, hidden = self.rnn1(x, hidden)
        x = x.select(0, maxlen-1).contiguous()
        x = x.view(-1, hidden_size)
        x = F.softmax(self.dense1(x))
        return x, hidden

    def init_hidden(self, batch_size=batch_size):
        weight = next(self.parameters()).data
        return Variable(weight.new(1, batch_size, hidden_size).zero_())

def var(x):
    x = Variable(x)
    if cuda:
        return x.cuda()
    else:
        return x

model = Net(features=features, cls_size=len(chars))
if cuda:
    model.cuda()
criterion = nn.CrossEntropyLoss()
optimizer = optim.Adam(model.parameters(), lr=lr)

def train():
    model.train()
    hidden = model.init_hidden()
    for epoch in range(len(sentences) // batch_size):
        X_batch = var(torch.FloatTensor(X[:, epoch*batch_size: (epoch+1)*batch_size, :]))
        y_batch = var(torch.LongTensor(y[epoch*batch_size: (epoch+1)*batch_size]))
        model.zero_grad()
        output, hidden = model(X_batch, var(hidden.data))
        loss = criterion(output, y_batch)
        loss.backward()
        optimizer.step()

for epoch in range(nb_epochs):
    train()
<ECODE> the input is “one-hot” vector and I tried changing its learning rate but the result is the same.
moskomule
I’m not sure, it’s hard to spot bugs in code that you can’t run. Why do you do this: <SCODE>x = x.select(0, maxlen-1).contiguous()
<ECODE> Don’t you want to return predictions for the whole sequence? It seems to me that you’re only taking the last output.
apaszke

return_sequences: Boolean. Whether to return the last output in the output sequence, or the full sequence.
moskomule
If you have the full code available somewhere I can take a look.
apaszke
OK, thanks. Does interfere in back propagation?
moskomule
How would they interfere? They both should be ok.
apaszke
moskomule
moskomule
apaszke
Right. So now, <SCODE>class Net(nn.module):
    ...
    def forward(self, x, hidden):
        x, hidden = self.rnn1(x, hidden)
        x = x.select(0, maxlen-1).contiguous()
        x = x.view(-1, hidden_size)
        x = F.relu(self.dense1(x))
        x = F.log_softmax(self.dense2(x))
        return x, hidden
...
criterion = nn.NLLLoss()
...
def train():
    model.train()
    hidden = model.init_hidden()
    for epoch in range(len(sentences) // batch_size):
        X_batch = var(torch.FloatTensor(X[:, epoch*batch_size: (epoch+1)*batch_size, :]))
        y_batch = var(torch.LongTensor(y[epoch*batch_size: (epoch+1)*batch_size]))
        model.zero_grad()
        output, hidden = model(X_batch, var_pair(hidden))
        loss = criterion(output, y_batch)
        loss.backward()
        optimizer.step()
<ECODE>
moskomule
apaszke
How to understand this fact? Thank you!
CodePothunter
Atcold
I see. Thank you very much!
CodePothunter
Hi, Thank you
osm3000