Hello!
I am using gradient clipping during training as follows:
optimizer.zero_grad()
loss = criterion(output, target)
loss.backward()
torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm = 1)
optimizer.step()

However, as you can see from the wandb plots
image1484×639 13.6 KB
Gradients explode, ranging from -3e5 to 3e5.
This plot shows the disribution of weights across each mini-batch. Thus, it is not an accumulated sum of all gradients.
I don’t know what is happening