Help debugging DenseNet model on CIFAR-10
Hi PyTorch community, -Brandon.
bamos
One thing is that this: <SCODE>for param_group in optimizer.state_dict()['param_groups']:
<ECODE> should be replaced with that: <SCODE>for param_group in optimizer.param_groups:
<ECODE> I know that the first version appears in the ImageNet example, but it no longer works as expected. I can’t see anything wront at a glance, but I’ll try to look more carefully into it sometime.
apaszke
Thanks for quickly looking! I fixed that. (It doesn’t sound like you expected this to help the convergence – the model’s still not converging as before.)
bamos
Thanks! My training code does successfully train a known model (VGGnet) on CIFAR-10, so my logic is also that there’s something wrong with my model. However I have also exactly compared my model’s outputs and gradients to the official model. I thought about trying to remove the passthrough connections from my DenseNet implementation to further debug this. However I haven’t tried this yet because I’ve never seen reports of such an architecture (even correctly implemented) converging on CIFAR-10. And if my implementation of this did converge, then it would indicate that there’s a problem of layers that concatenate the input and output. So to directly check if there’s a problem with this kind of operation, I used numdifftools to numerically check the gradients of a single PyTorch layer that concatenated the input to a fully-connected operation. As another idea of breaking the DenseNet into a known architecture, I could start with a ResNet architecture that’s known to converge and then start adding DenseNet features. However these intermediate architecture are not known to converge, so if it doesn’t work, then I won’t know if it’s because of a code bug or something more fundamental.
bamos
Adam’s been helping me debug this over Slack today and we’ve solved it! We found a new PyTorch bug with cudnn that comes up with DenseNet-style layers. After fixing this, my DenseNet model converges much better than before, and I’ll update my repo with the current results shortly. My understanding is that Adam will push a short patch to PyTorch master soon. Thanks again for the help, Adam. -Brandon.
bamos
The fix is already in master.
apaszke
What was the problem ? Been having some convergence issues with my network which has some layers concatenation just as yours. My problems are not necessarily related to yours, but I’m glad you could figure it out with adam !
ClementPinard
The problem arised form concatenating outputs of convolution along second dimension. This lead to calling conv’s backward with non-contiguous gradients, and we were overly smart about reusing cuDNN descriptors, so the backend thought that it’s contiguous. The fix is to either disable cuDNN or rebuild pytorch.
apaszke
fmassa