Hi Alex.
My batch size is set to 32, which is pretty small. I’m running on a Titan X which has about 12GB of VRAM. Originally I’m trying to implement a simple language model so all my inputs are text.
Here is all the tensor object returned from python gc module, printed in type, shape, size(MB) format
[(<class 'torch.nn.parameter.Parameter'>, (17794, 300), 21.3528), 
(<class 'torch.nn.parameter.Parameter'>, (17794, 300), 21.3528), 
(<class 'torch.Tensor'>, (17794, 300), 21.3528), 
(<class 'torch.Tensor'>, (17794, 300), 21.3528),
(<class 'torch.Tensor'>, (17794, 300), 21.3528), 
(<class 'torch.Tensor'>, (17794, 300), 21.3528), 
(<class 'torch.nn.parameter.Parameter'>, (17794,), 0.071176),
(<class 'torch.Tensor'>, (17794,), 0.071176), 
(<class 'torch.Tensor'>, (17794,), 0.071176), 
(<class 'torch.Tensor'>, (1824, 17794), 129.825024),
(<class 'torch.Tensor'>, (1824,), 0.007296), 
(<class 'torch.Tensor'>, (57, 32, 17794), 129.825024),
(<class 'torch.Tensor'>, (57, 32), 0.007296), 
(<class 'torch.Tensor'>, (57, 32), 0.007296), 
(<class 'torch.Tensor'>, (57, 32), 0.007296), 
(<class 'torch.Tensor'>, (32,), 0.000128),
(<class 'torch.Tensor'>, (), 4e-06)]