Update:
I discovered the part which was causing the problem in the above code as marked. “toep” is a list of numpy objects. In order to use it in my code, I had to convert its elements to cuda objects. I could not find an elegant way to do this so I just looped over it. Now this variable, for some reason, was accumulating GPU storage and not releasing it back. I tried deleting it but it did not work.
.
.
	    #**********************  PROBLEMATIC PART **************************************
	    .
	    .
	    .
	    for j in range(len(toep)):     # <~~ This part was accumulating 
		    toep[j] = toep[j].cuda()   # <~~ GPU memory
	    .
	    .
	    . 
	    .
	    .
	    #*******************************************************************************
.
.

I could not find an elegant solution so what I found worked is by taking this variable back to the CPU.
 .
 .
	    #**********************  PROBLEMATIC PART **************************************
	    .
	    .
	    .
	    for j in range(len(toep)):     # <~~ This part was accumulating 
		    toep[j] = toep[j].cuda()   # <~~ GPU memory
	    .
	    .
	    for j in range(len(toep)):     # <~~ I solved it by  
		    toep[j] = toep[j].cpu()  # <~~ taking it back to the cpu
	    .
	    .
	    #*******************************************************************************
.
.

However, I still have questions left. Why was this variable accumulating GPU memory? And please let me know if there is an better alternative solution to achieve the same task.