Hello,
thank you for pytorch! Would you have a hint how to approach ever increasing memory use?
While playing around with the (very cool, thanks Sean!) deepspeech.pytorch 26 , I notice that the (RAM, but not GPU) memory increases from one epoch to the next.
Being on python 3.5 I used tracemalloc to see where memory is allocated. To my surprise, I am seeing the forward pass (“out = model(inputs)”) as one of the top allocators for each cycle.
Is it expected that the forward pass accumulates memory?
I there a standard way how to find out where exactly that happened (I have pasted  the backtrace of the main allocation that comes up when comparing allocation between to epochs below)?
I would appreciate any hint or pointer.
Thank you!
Thomas
P.S.: I have adapted the script to set a few defaults differently and use a different dataset, but did not do much with the actual calculation (and I’d be happy to provide my changes if they are of interest). I am using a git checkout of github pytorch master from today.
  File "/usr/local/lib/python3.5/dist-packages/torch/backends/cudnn/__init__.py", line 387
    return array_type(*itr)
  File "/usr/local/lib/python3.5/dist-packages/torch/backends/cudnn/__init__.py", line 160
    int_array(tensor.size()), int_array(tensor.stride())))
  File "/usr/local/lib/python3.5/dist-packages/torch/backends/cudnn/__init__.py", line 398
    descriptor.set(tensor)
  File "/usr/local/lib/python3.5/dist-packages/torch/backends/cudnn/rnn.py", line 242
    fn.cy_desc = cudnn.descriptor(cx) if cx is not None else None
  File "/usr/local/lib/python3.5/dist-packages/torch/nn/_functions/rnn.py", line 269
    cudnn.rnn.forward(self, input, hx, weight, output, hy)
  File "/usr/local/lib/python3.5/dist-packages/torch/autograd/function.py", line 224
    result = self.forward_extended(*nested_tensors)
  File "/usr/local/lib/python3.5/dist-packages/torch/autograd/function.py", line 202
    flat_output = super(NestedIOFunction, self)._do_forward(*flat_input)
  File "/usr/local/lib/python3.5/dist-packages/torch/nn/_functions/rnn.py", line 327
    return func(input, *fargs, **fkwargs)
  File "/usr/local/lib/python3.5/dist-packages/torch/nn/modules/rnn.py", line 91
    output, hidden = func(input, self.all_weights, hx)
  File "/usr/local/lib/python3.5/dist-packages/torch/nn/modules/module.py", line 202
    result = self.forward(*input, **kwargs)
  File "deepspeech.pytorch/model.py", line 48
    x, _ = self.rnn(x)
  File "/usr/local/lib/python3.5/dist-packages/torch/nn/modules/module.py", line 202
    result = self.forward(*input, **kwargs)
  File "/usr/local/lib/python3.5/dist-packages/torch/nn/modules/container.py", line 64
    input = module(input)
  File "/usr/local/lib/python3.5/dist-packages/torch/nn/modules/module.py", line 202
    result = self.forward(*input, **kwargs)
  File "deepspeech.pytorch/model.py", line 94
    x = self.rnns(x)
  File "/usr/local/lib/python3.5/dist-packages/torch/nn/modules/module.py", line 202
    result = self.forward(*input, **kwargs)
  File "/usr/local/lib/python3.5/dist-packages/torch/nn/parallel/data_parallel.py", line 59
    return self.module(*inputs[0], **kwargs[0])
  File "/usr/local/lib/python3.5/dist-packages/torch/nn/modules/module.py", line 202
    result = self.forward(*input, **kwargs)
  File "train_de.py", line 139
    out = model(inputs)