PyTorch vs Theano GPU RAM usage issue
Hello, I’ve noticed that PyTorch (torch (0.1.10.post2), torchvision (0.1.7)) uses significantly more GPU RAM than e.g. Theano running a similar code. Unfortunately I cannot disclose the actual code but I think it should be possible to reproduce this behavior with the following simple sequential architecture (all dimensions are in form MINIBATCH_LENGTH x NUM_CHANNELS x H x W): N - minibatch size, e.g. 20, padding mode == “same” everywhere Main loop: <SCODE>net = Net()
net.cuda()
net.train()

optimizer = optim.SGD(net.parameters(), lr = ...)
criterion = nn.MSELoss()

input = Variable(torch.from_numpy(<your-Nx10x1024x1024-tensor>).cuda())
target = Variable(torch.from_numpy(<your-Nx10x1024x1024-tensor>).cuda())

for epoch in xrange(...):

    output = net(input)
    loss = criterion(output, target)
    
    net.zero_grad()
    loss.backward()
    optimizer.step()
<ECODE> On a GTX-1060 corresponding code takes around 30% more GPU RAM than its Theano counterpart (exection times are about the same for Theano and PyTorch versions). Is this something that can be fixed? Thanks,
vladimir
This might be one reason to explain the memory difference.
smth
thanks for your reply. This seems to be the case indeed (the maximum mini-batch size seems to be quite similar for both Theano and PyTorch). However, is there a way to constrain memory allocation? In Theano there’s lib.cnmem which can be assigned % of memory to pre-allocate. It’s a soft limit but still helpful when sharing GPU between multiple jobs.
vladimir
there is no way to constrain the memory to an upper bound right now.
smth