Hi everyone.
Here is my problem :
I’ve been playing around with some neural networks lately, trying to compare different kinds of normalization layers on a simple problem (namely VGG16 on the CIFAR10 dataset).
However the loss graph for only one model is really noisy, so I decided to plot a mean of the losses for 5 runs of the same architecture. Let L_it be the loss for the i-th model on the minibatch at time t, each of my point p_t is computed by the following :

I’m also plotting a rolling loss to smooth everything a bit, computed as follows :

And here is the kind of result that I get :

Batch_Norm1170×876 65.8 KB

The steps which are appearing between every epochs shouldn’t be there I guess… I talked to my teachers about it, they all agree that it isn’t normal, but none of them has been able to find the problem.
Concerning the code I use to produce those results :
I really tried to keep it as simple as possible, and it follows really closely the code provided in the Pytorch tutorial for CIFAR10.
My DataLoader is created with this code :
train_transform = transforms.Compose([transforms.RandomHorizontalFlip(),
            transforms.ToTensor(),
            transforms.Normalize((.5, .5, .5), (.5, .5, .5))])

trainset = torchvision.datasets.CIFAR10(root = "../data_CIFAR10", train = True, download = False, transform = train_transform)
trainLoader = torch.utils.data.DataLoader(trainset, batch_size = bs, shuffle = True, drop_last = True)

My model, the criterion and the optimizer are here :
model = models.vgg16_bn(num_classes = 10)
model.to(gpu)
criterion = nn.CrossEntropyLoss()
optimizer = optim.SGD(model.parameters(), lr = 0.003)
train_losses = train_model(model, trainLoader, criterion, optimizer, nb_epochs, gpu)

And finally, here is the `train_model function :
def train_model(model, trainLoader, criterion, optimizer, epochs, gpu):
    train_losses = []
    for epoch in range(epochs):
        for i, data in enumerate(trainLoader, 0):
            inputs, labels = data[0].to(gpu), data[1].to(gpu)

            optimizer.zero_grad()
            outputs = model(inputs)
            loss = criterion(outputs, labels)
            loss.backward()
            optimizer.step()

            train_losses.append(loss.item())

            if(i%100==99):
                print("[%d, %5d] loss : %.5f" % (epoch + 1, i + 1, loss.item()))

    print("Finished training\n\n")
    return train_losses


Did I miss anything obvious? I tried to describe my problem as precisely as possible (while keeping it simple), but if you need more information please ask.
Thanks.
(PS : what’s the best way to integrate LaTeX equations in a post?)