Hi there, I’m running some profiling on my training code and I realized that every 11 iterations (batches) the code freezes for a while when calling optim.step then it resumes normally for the next 11 iterations. Does it have to do with gradient checking or maybe a bug in my code?
The training loop is as follows:
for i, (index, im, mask) in tqdm(enumerate(train_loader), total=n_iter):
    self.optimizer.zero_grad()
    im = im.cuda()
    mask = mask.cuda()
    # Forward propagation
    logit = self.forward(im)
    loss = self.criterion(logit, mask)

    # Compute metrics
    if i % metric_step == 0:
        pred = torch.sigmoid(logit)
        iou  = eval.dice_accuracy(pred.data.cpu().numpy(),
                                  mask.cpu().numpy(),
                                  is_average=False)

        train_loss.append(loss.item())
        train_iou.extend(iou)
    loss.backward()
    self.optimizer.step()  # <- freezes here on iterations multiples of 11

Many thanks.
Kind regards,