Hi,
I have an NN model which is written from stratch.This model has 2 neurons input layer, 1 hidden layer with 5 neurons and 2 output neurons. In hidden layer there is tanh activation. Cross entropy loss is being used. Weight 2 regularization aplied with lambda 0.01. Also weights are initialized with Xavier initialization. Learning rate is: 0.001. After 1000 epoch, this model has gives a Loss value of 0.093, and accuracy of 0.970 with below dataset.
x,y = sklearn.datasets.make_moons(n_samples=200,noise=0.05)
I decided to build the same model using Pytorch. However, I got very poor results in Pytorch. It seemed to be quiet strange.
Can you please check my Pytorch code and comment on why did I got poor results?
input_neurons = 2
hidden_neurons = 5
output_neurons = 2
learning_rate = 0.001
lambda_reg = 0.01
x,y = sklearn.datasets.make_moons(n_samples=200,noise=0.05)
x = torch.FloatTensor(x)
y = torch.LongTensor(y)
class FeedForward(torch.nn.Module):
def init(self, input_neurons, hidden_neurons, output_neurons):
super(FeedForward,self).init()
self.hidden = nn.Linear(input_neurons, hidden_neurons)
self.out = nn.Linear(hidden_neurons,output_neurons)
def forward(self, x):
x = self.hidden(x)
x = F.tanh(x)
x = self.out(x)
return x
def init_weights(m):
if type(m) == nn.Linear:
torch.nn.init.xavier_uniform(m.weight)
#m.bias.data.fill_(0.01)
network = FeedForward(input_neurons = input_neurons, hidden_neurons = hidden_neurons, output_neurons = output_neurons)
network.apply(init_weights)
optimizer = torch.optim.SGD(network.parameters(), lr = learning_rate,weight_decay=lambda_reg)
loss_function = torch.nn.CrossEntropyLoss()
for epoch in range(1000):
out = network(x)
loss = loss_function(out, y)
optimizer.zero_grad()
loss.backward()
optimizer.step()
if epoch % 50 == 0:
    max_value, prediction = torch.max(out, 1)
    predicted_y = prediction.data.numpy()
    target_y = y.data.numpy()
    accuracy = (predicted_y == target_y).sum() / target_y.size
    print('Accuracy = {:.2f} Loss = {:.2f}'.format(accuracy,loss))