Thanks a lot for the reference to the memory and cpu usage methods.
After deleting the “requires_grad = True”, the program now runs for batch_size=1 (with 2 GB of 4 GB RAM used) but get stuck in the second iteration when I use batch_size = 2. Using the method cpuStats() before and after the line optimizer.step() shows that it still uses 2 GB of GPU RAM, but get “out of memory” during the  optimizer.step() call in the second iteration, with the error reported as:
    denom = exp_avg_sq.sqrt().add_(group['eps'])
RuntimeError: cuda runtime error (2) : out of memory at c:\programdata\miniconda3\conda-bld\pytorch_1524546371102\work\aten\src\thc\generic/THCStorage.cu:58

Changing the optimizer to optim.SGD (i.e. defining optimizer = optim.SGD(model.parameters() , lr=0.001)) indeed allows me to run through the program with a batch size of 10! It also seems like the memory usage stays around 2 GB, even when I increase the batch size even more…
Is there any way to use Adam algorithm without getting “out of memory” in my case?
One more question - when I am not specifying “requires_grad = True”, how does the line   loss.backward() work (where loss = criterion(outputs, label)) ? shouldn’t I put requires_grad=True for at least one of the variables outputs, or label? (or, use “with torch.set_grad_enabled”)
Thanks a lot!