I just removed the AverageMeter and store the instant values. Here is the complete code: https://gist.github.com/zeakey/9d1c313329a7ea32ea12ae0f3a8db09f#file-train_center_exclusve_filter-py 10.
Strangely still the loss (and top1 accuracy) vibrates in a zigzag manner, the first batch always evaluates the highest accuracy and lowest loss.