I’ve looked through the docs and forums, but haven’t figured out why the below code is leaking memory during training. It does not leak when calling forward, and I am deleting all intermediate variables + calling gc.collect() + calling torch.cuda.empty_cache(), but still accumulate 5GB of garbage after the first batch.

  
      gist.github.com
  
  
    https://gist.github.com/tbenst/476f26dc96c0ab2d9b0d6c13b9fa6d43 4
mem_leak.ipynb
{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch as T\n",
    "import torch\n",
This file has been truncated. show original




  
  
    
    
  
  


Any tips on how to further troubleshoot? The above notebook should be fully reproducible and uses random data. Many thanks!