Hi. I’m using a PyTorch Hook to analyzing intermediate feature map’s statistics with a large-scale model training.
Also I’m using multi-GPU to accelerate the training process.(with torch.nn.DataParallel)
The problem is, GPU utilization is drastically drops down when my Hook is executing(I compared GPU utilization with when I don’t activated my Hook - naive training without analyzing)
I know this is Indispensable situation because my analyzing Hook is quite high-cost function, but all my multi-GPU are freezing in Hook phase, which seems not reasonable.
I think there’s some solution, such as Hook-specify GPU setting, like:
GPU 0~3: model training
GPU 4: gather intermediate feature map and analyze
Is it possible? Or is there any good solution to solve my problem?
Any answers will be welcomed. Thanks.