I am also intrigued to find so many python processes resulting from the above command:
Screen Shot 2021-01-06 at 10.11.28 AM1138×832 41.6 KB
Also, memory usage shows up ~30 gigs against each process. My teacher model is a t5-large with 770 million parameters. That should be about 3 gigs for parameters assuming 32-bit, but in this case fp16 is turned on.The student model is down to 6 layers from teacher’s 24 layers, so roughly 1/3 the size. Batch size is 2. I am not sure what is accounting for the ~30 gigs. Following is the GPU usage-
Screen Shot 2021-01-06 at 10.34.40 AM1120×656 32 KB
GPU memory usage is pretty low.
I am probably missing some things. Any thoughts would be appreciated!
Also I still can’t get the learning curves to show up. I forced relogin with wandb and tried using a new API key but that didn’t help.
Thanks,
Sumithra