Having the same issue here: process PSS consumption increases twice after a single call  with torch.no_grad(): model(torch.rand((1, 3, 400, 400)))
 PID User     Command                         Swap      USS      PSS      RSS                                                 
>> python
16346 lilyai   python                             0     3536     3658     8844
>> import torch
16346 lilyai   python                             0    45440    47373    99584
>> model = torch.jit.load
>> model.eval()
16346 lilyai   python                             0   205188   207152   260088
>> with torch.no_grad(): model(torch.rand((1, 3, 400, 400)))
16346 lilyai   python                             0   433532   435863   495292