Thanks so much Adam! 1+2 answers my questions completely. Regarding 3 I realized I asked two questions in one.
The automatic management (and trying to avoid cudaMalloc and cudaFree) makes sense. With my training loop the GPU memory does grow, but eventually stops once the data has been exhausted. That matches what you describe.
What I don’t understand is that when I’m sampling an RNN, the memory grows without stopping. So the following code will work for (for example) length=100 but will erratically quit around length=200 in my case:
def evaluate(model, seeds, length=100):
    seeds = torch.from_numpy(seeds).cuda()
    hidden = model.create_hidden(batch_size=1).cuda()
    hidden = Variable(hidden)

    # prime model with all but last element
    for seed in seeds[-2:]:
        x = Variable(seed.view(1, 1, -1))
        _, hidden = model(x, hidden)

    # run final element and continue
    x = Variable(seeds[-1])
    predicted = []
    for i in range(length):
        x, hidden = model(x.view(1, 1, -1), hidden)
        predicted.append(x.data.cpu().numpy().reshape(-1))

    return np.asarray(predicted)

The GPU memory is released when the function returns, which makes me think that I’m doing something wrong with the way I use x or hidden. Is this my mistake, or does it sound more like a bug? If it sounds like a bug I can try to produce a minimal complete chunk of code that reproduces it.