I’m working with RNNs for medium-sized data (fits on a single machine, probably won’t need multiple GPUs).
But I want to get the most performance out of my RNN with the GPU I have, so I’ve been testing with even smaller datasets to make sure I understand the principles behind moving memory around with pytorch. I want to check my understanding to see what I’m missing. I have experience with graphics programming (OpenGL shaders, frame/vertex/pixel buffers and streaming, etc) but not really with CUDA.
My GPU memory is going to be divided mainly between the model parameters and the batch. Let’s say I want my model to be big, so I use about 70% of the GPU memory for the model and optimizer parameters. Then the batch might take 10% of the memory.
While processing one batch I do this:

Access the data (stored in RAM as a torch.tensor). Each batch has size (seq_length+1, batch_size).
Move the tensor to GPU, converting it to a torch.cuda.tensor with batch_tensor = batch_tensor.cuda()

Create two variables from sliced chunks of the tensor: Variable(batch_tensor[:-1]) and Variable(batch_tensor[1:])

Reset the model, run prediction, loss, backprop, and optimization.

Questions:

If I can fit two batches in memory at the same time, can I be swapping one out while the other is still processing? How can I do this?
Is the slicing operation slow/doing copying, or is it fast/more like reshaping?
When does a torch.cuda.tensor get unallocated from the GPU? It looks like running a training loop steadily grows my GPU memory usage with every cuda() call, and it’s never freed up. Is there a way to generate an indefinitely long sequence with a fixed amount of GPU memory?
Are there any other optimizations I can make?

For completeness, here is a test notebook on a small dataset (char-rnn/shakespeare) showing what I’ve got so far. I would also really appreciate any other general feedback about the code.

  
      gist.github.com
  
  
    https://gist.github.com/kylemcdonald/2d06dc736789f0b329e11d504e8dee9f 40
pytorch+char-rnn.ipynb
{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
This file has been truncated. show original