I have finally figured out the problem in my code. In my test() function I modified the line from,
patches, counts = Variable(patches), Variable(counts)
to
patches, counts = Variable(patches, volatile=True), Variable(counts)
Adding this attribute does the trick and GPU memory utilization is no longer growing.