Thank you for your reply. However, my accuracy and loss did not change much in the whole learning process. I have tried changing the optimizer to Adam optimizer, learning rate, but the accuracy and loss still do not change.
Any idea where I can start inspecting?
This is how I calculated accuracy and loss
for (inputs, labels) in test_loader:
        inputs, labels = inputs.to(device).float(), labels.to(device)
        inputs = inputs.view(-1, 6, 1, 30)        
        labels = labels[:, -1].long()
        labels = labels - 1
        optimizer.zero_grad()
        outputs = net(inputs)
        loss = criterion(outputs, labels)
        sum_loss += loss.item()
        predicted = torch.argmax(outputs, dim=1)
        sum_total += labels.size(0)
        sum_correct += (predicted == labels).sum().item()
        print("test  mean loss={}, accuracy={}"
            .format(sum_loss*BATCH_SIZE/len(test_loader.dataset), float(sum_correct/sum_total)))
    test_loss_value.append(sum_loss*BATCH_SIZE/len(test_loader.dataset))
    test_acc_value.append(float(sum_correct/sum_total))

Accuracy:
Screen Shot 2020-10-13 at 19.21.04828×778 32 KB