I’m transferring a Caffe network into PyTorch. However, when I’m training the network with exactly same protocol, the training loss behaves like this:

untitled.png987×634 20 KB

The loss increasing within each epoch and decreases when starting a new epoch. Thus forms this sawtooth-shaped loss.
Two problems:

The increasing of loss within each epoch seems to be the problem with momentum. I set the momentum to 0 (originally 0.9 by Caffe protocol) and the shape goes away. Is there any difference between Caffe and PyTorch momentum setting?
Let’s assume the problem is with momentum, the loss should not decrease at the start of each epoch, i.e. if the momentum is too large, the loss will always increase. Is there a hidden cleanup operation at the start of each epoch?

Here’s my code:
net.train()
for epoch in range(1, args.epochs + 1):
    net.train_step(epoch, log_interval=10)

and in class Net:
def train_step(self, epoch, log_interval=100):
    for batch_idx, (data, target) in enumerate(self.train_loader):
        if self.is_cuda:
            data, target = data.cuda(), target.cuda()
        data, target = Variable(data), Variable(target)
        self.optimizer.zero_grad()
        loss = self.forward(data, y=target)
        loss.backward()
        self.optimizer.step()
        if batch_idx % log_interval == 0:
            print(
                Train Epoch: {} [{}/{} ({:.0f}%)]\t'.format(epoch, batch_idx * len(data),
                                                             len(self.train_loader.dataset),
                                                             100. * batch_idx / len(self.train_loader))
                + '\t'.join(
                    ['{}: {:.6f}'.format(key, self.loss.loss_value[key].data[0]) for key in
                     self.loss.loss_value.keys()]))

Thanks!