Per your ealier point, the graph is building on itself and continuing to increase in size. It was subtle detail for me as a newbie to PyTorch.
I am using an RNN with LSTM units.  I have to detach the hidden states at each pass otherwise I build up the size of the network.
From an example, I had already gotten:
def repackage_hidden(h): """Wraps hidden states in new Variables, to detach them from their history.""" if type(h) == Variable: return Variable(h.data) else: return tuple(repackage_hidden(v) for v in h)
but I was not calling that on my validation loop.  Adding that in, I can proceed and validation works with no trouble.