github.com
  
  
    pytorch/examples/blob/f83508117b1ba9b752b227de992799093af3b215/imagenet/main.py#L193 10



end = time.time()
for i, (input, target) in enumerate(train_loader):
    # measure data loading time
    data_time.update(time.time() - end)


    target = target.cuda(non_blocking=True)


    # compute output
    output = model(input)
    loss = criterion(output, target)


    # measure accuracy and record loss
    prec1, prec5 = accuracy(output, target, topk=(1, 5))
    losses.update(loss.item(), input.size(0))
    top1.update(prec1[0], input.size(0))
    top5.update(prec5[0], input.size(0))


    # compute gradient and do SGD step
    optimizer.zero_grad()
    loss.backward()



  
  
    
    
  
  


This issue is with most of the implementations, but for you can for example check out the pytorch imagenet implementation. Upon digging further I found that we can give reduce=False as an argument to the loss function, which will prevent averaging out the loss.
However, the question remains as to whether the same thing happens (by a way of default) in case of hidden layers during back-propagation?