Dear ptrblck,
Thank you for replying. After you said that the GPU usage should not change I commented out the problematic part as shown below, and indeed there was no change.
Due to the nature of the code, and the dataset dependency I feel it would be very difficult for me to provide you with an executable snippet. Part of the reason is that there are various dependencies that the net is using.
What the code is doing is first apply a CNN network to the input followed by conjugate gradient steps on the CNN output. This conjugate gradient step needs an operator which is “EncObj”. To make this operator we require csm and toep which is dependent on the input data.
class net(nn.Module):

    def __init__(self, EncObj, CNN, ..other parameters..):

	    self.cnn = CNN
	    self.EncObj = EncObj
	    self.opcomp = None

    def ConjGrad(self, x, ..other parameters..):		
	    .
	    x = self.EncObj.apply_AdagA_Toeplitz(x)
	    .
	    .
	
    def forward(self, x):
	
	    # apply the CNN;
	    xm = self.cnn(xp)	

	    #**********************  PROBLEMATIC PART **************************************
	    # perform conjugate gradient on the cnn output
	    csm = self.opcomp[0].cuda()
	    toep = self.opcomp[1]  # This is a list of numpy objects which needs to be put on the GPU
	    for j in range(len(toep)):
		    toep[j] = toep[j].cuda()
	    self.EncObj.NUFFT = MriSenseNufft(smap=csm, ..other parameters..)  # an imported function 
	    self.EncObj.AdjNUFFT = AdjMriSenseNufft(smap=csm, ..other parameters..)  # an imported function 
	    self.EncObj.ToepNUFFT = ToepSenseNufft(csm)  # an imported function 
	    self.EncObj.AdagA_toep_kernel_list = toep
	    xm = self.ConjGrad(xm, ..other parameters..)
	    #*******************************************************************************

	    return xm

# Calling the net from some other evaluating code
.
with torch.no_grad():
    for index in range(Ids ):
        bx = x[index,:]    # the input
        by = y[index,:]    # the labels
        bop = op[index,:]  # data dependent operator
        net.opcomp = bop
        l = loss(net(bx.detach.()),by)
        .
        .
    .
    .
.
.

I tried deleting the variables csm and toep after performing the conjugate gradient steps because we do not need it anymore until the next iteration where a new csm and toep initialized anyways. It did not help and this problematic snippet kept on accumulating GPU memory. If there is a blatant mistake I am doing please let me know.