When running a tiramisu model, I found that the gpu use was 4.5 GB during 1st epoch and during the 2nd it shot up. Based on your comment I did the following
out = model(input)
loss = crite(out, labels)
loss.backward()
del loss
del out
And thankfully I saw my network using the same amount of GPU in the consecutive epochs