It is possible that the gpu memory increases a bit until the second or third forward passes. If the memory keeps increasing steadily after that, that means that your code is doing something wrong 
Could you give us a minimal sample to reproduce this so that we can check?