I am glad to help you!
I think it is a feature in Python, and you could run your networks in practice and to see if there is a GPU memory wastage or not, if our conclusion is wrong please correct me.
Here is the thread on stackoverflow 17 which helped me a lot few days ago.