rcshubhadeep:

Another update, some experiments with lr (0.01) and my first model shows that sometimes The loss does decrease. Sometimes not. That can even happen in two consecutive runs. This does not make sense to me. Why the behavior is not consistent if everything else stays the same? What am I missing here?


This might be expected, if your overall training is unstable and you would have to play around with some hyperparameters. Also, if you don’t seed the code and don’t force it to be deterministic, slight variations are expected. In the best case your training would be stable enough to yield approx. the same final performance. However, as your loss curves show, the training gets often stuck.
The model is generally able to overfit random data as seen here:
torch.manual_seed(2809)

model_conv = nn.Sequential(nn.Conv2d(3, 8, kernel_size=3, padding=1), nn.ReLU(),
                           nn.AvgPool2d(kernel_size=2, stride=2),
                           nn.Conv2d(8, 16, kernel_size=3, padding=1), nn.ReLU(),
                           nn.AvgPool2d(kernel_size=2, stride=2),
                           nn.Conv2d(16, 32, kernel_size=3, padding=1), nn.ReLU(),
                           nn.AvgPool2d(kernel_size=2, stride=2),
                           nn.Conv2d(32, 64, kernel_size=3, padding=1), nn.ReLU(),
                           nn.AvgPool2d(kernel_size=2, stride=2),
                           nn.Flatten(),
                           nn.Linear(3136, 256), nn.ReLU(),
                           nn.Linear(256, 128), nn.ReLU(),
                           nn.Linear(128, 84), nn.ReLU(),
                           nn.Linear(84, 2)
                          )

data = torch.randn(10, 3, 120, 120)
target = torch.randint(0, 2, (10,))
optimizer = torch.optim.Adam(model_conv.parameters(), lr=1e-3)
criterion = nn.CrossEntropyLoss()

nb_epochs = 200
for epoch in range(nb_epochs):
    optimizer.zero_grad()
    output = model_conv(data)
    loss = criterion(output, target)
    loss.backward()
    optimizer.step()
    print('epoch {}, loss {}'.format(epoch, loss.item()))