You don’t need to use float(loss.item()). .item() returns a standard python float and thus won’t be tracking the computation graph.
How is your GPU memory usage in the first epochs? Are you already approx. on the limit?
Could you observe the memory with nvidia-smi -l and see if the memory usage is increasing in the first epochs?