Hi,
correct me if I am wrong. I think the large amount of GPU memory is used up by the training data, the network parameters are those comparisonly small to the data loaded to GPU memory for training.
you might try to reduce the batch size by half and see whether this situation improve?
Thanks.
rgds,
CL