Convergence speed difference compared to that of Torch7
I implemented the same autoencoders with BCE criterion using pytorch and torch7. When I tried to train them using SGD with momentum, the convergence speeds were almost same within about 2000 iterations. But, the convergence of pytorch becomes quite slower than that of torch7. Has anyone compared the convergence performances ?
Seungyoung_Park
smth
I also tried rmsprop with batch normalization, the same things were observed. The below is a modified rmsprop employed in Torch7. <SCODE>square_avg:mul(alpha)
square_avg:addcmul(1.0-alpha, grad, grad)
avg:sqrt(square_avg+eps)
params:mul(1-lr*weight_decay):addcdiv(-lr, grad, avg)
<ECODE> And, the below is an equivalent rmsprop in pytorch <SCODE>square_avg.mul_(alpha).addcmul_(1 - alpha, grad, grad)
avg = square_avg.add(group['eps']).sqrt()
p.data.mul_(1-group['lr']*weight_decay).addcdiv_(-group['lr'], grad, avg)
<ECODE> Are they equivalent ? Also, my pytorch code didn’t work at all before 0.19 version due to some run time errors. Is that related to this convergence difference ?
Seungyoung_Park
apaszke
Thanks for your reply. Is it possible that the convergence becomes slower even if some gradients becomes NaNs or infs ? My model is just autoencoder consisting of convolution, maxpooling, maxunpooling, relu, and sigmoid. Anyway, to compare the gradients, I found that the gradients can be saved in a flatted form by <SCODE>torch.cat([g.grad.view(-1) for g in model.parameters()],0)<ECODE>
Seungyoung_Park
apaszke