with torch.no_grad() does help, thanks a lot! But I still want to leave this thread open to see if there are other solutions without freezing the weights, as there is little difference of max memory use of a single card between spliting extractor into 2 and 3 parts.