Hi,
I’m monitoring my GPU usage as I’m doing training.
What I observe is, it fluctuates between 93% and 44%.  Is this because, at some points, it waits CPU to pass data? Or is it because I do a pass over validation set (which is considerably smaller than the training set) after training ?
In any case, does these fluctuations mean I’m under-utilizing my GPU? If that’s the case, what I am doing wrong? For example, I tried to increase  num_workers on my data loaders but it didnt have any effect on the usage pattern.