I realized new experiments and it seems that my issue is somehow related to APEX!
When I do not use apex.amp, the GPU memory increases at the end of the first loop but then remains constant (489 MB before the first iteration and then 807MB at the beginning of all other iterations)
On the other hand, when I use apex.amp, the GPU memory seems to increase during the first iterations to some point (between 900MB and 1GB) and then varies in this interval.
Then my two questions are:

why does the GPU memory usage vary from one iteration to the other?
why the GPU memory usage is higher using apex than using standard PyTorch?

I don’t know to what extent it is still relevant to ask such question in this forum since it seems to be not directly related to PyTorch. Please tell me if you prefer me to close that topic and open an issue directly on apex repo.