I see the same problem.  The train works fine, but as I try to run validation data the memory on the card increases till I get a CUDA out of memory error.  Very interested in a solution to this.
FWIW, my model is an RNN with LSTM units.  Not sure if that matters.