Hi ptrblck
Thank you for the reply. All right that makes sense.
My batch size is 1 (i.e. single images are passed forward). My understanding of the implementation of running mean and running variance is that it remembers the statistics of the previous batches and adjusts the statistics for the normalization layers as it goes through batches (i.e. in my case the single training images). This means that a batch size of 1 is not a problem.
Can you briefly state if this is correctly understood?
“defnet” is an import of a script with my network as well as this training function:
def train(network, image, target):
    criterion = nn.CrossEntropyLoss()
    optimizer = ts.optim.Adam(network.parameters())
     
    # variable wrap
    image, target = ts.autograd.Variable(image), ts.autograd.Variable(target)
        
    # forwardpass, backprop, optimization step
    prediction = network(image)
    optimizer.zero_grad()
    loss = criterion(prediction, target)
    rloss = loss.item()
    loss.backward()
    optimizer.step()
        
    return network, rloss

I would think that this would work, but I don’t know if there is a more efficient way around training.
The training function by the way returns loss.item() to answer your last question.
Best regards,
Simon