kendreaditya:

I tried adding those CNN layers and the model did a bit worse averaging around an accuracy of ~45% on the test set. I tried manually scheduling the learning rate every 10 epochs, the results were the same. Weight decay didn’t seem to effect the results much, changed it from 0.1-0.000001.
From previous testing, I have a model that achieves 60% acc on both the testing and the training set. However, when I try to retrain it, the acc instantly goes down to ~40 on both sets (training and testing), which makes no sense. I have tried altering the learning rate from 0.01 to 0.00000001, and also tried weight decay for this.
From training the model and the graphs, it seems like the model dosn’t know what it’s doing for the first 5-10 epochs and then starts to learn rapidly to around ~50%-60% acc on both sets. This is where the model starts to overfit, form there the model’s acc increases to 100% on the training set, and the acc for the testing set goes down to 33%, which is equivalent to guessing.


The results do make sense the loss at least. Depending on the type of testing, I either use code that logs the data or just trains the model directly. Here is my simplified code (that I use and have found no errors or bugs in):
for epoch in t:  # loop over the dataset multiple times
    for i, data in enumerate(Htrainloader, 0):
        # get the inputs; data is a list of [inputs, labels]
        inputs, labels = data
        inputs = inputs.to(device)
        labels = (labels.to(device))

        optimizer.zero_grad()

        outputs = model(inputs)
        loss = criterion(outputs, labels)
        loss.backward()
        optimizer.step()

Using this code with the check you suggested, I got the same results for accuracy, meaning the code is working correctly.
Note: this code is different than the code that was used to produce the image int the original post.
Anything else I can try? Other hyperparameter options?