This would mean that additional tensors are most likely pushed or created on the device, so compare the memory usage before resetting the states and after each step of loading the model to isolate where the memory overhead is coming from.