Sorry, what I wrote in the previous post was not clear. The GPU usage keeps growing while you are training not necessarily having a relation to the epochs. I took a closer look at the GPU memory usage. On worker #1, I kept using
nvidia-smi

while the program is running and the memory usage kept increasing. The original GPU memory usage should be small and around 500MB. But if I use the RPC it will keep growing, and every second I typed nvidia-smi and I can see a few MB increased and if it is between epochs, I can also see a big jump of increment of memory usage.
Best,
Ziyi