Projects could be really complicated. To understand what is a problem I would try to profile my program and find the bottleneck. It could be in any major moving part of the pipeline: from preparing data and generating the batches to training the models on the distributed system. So, for example I would check if my cpus are struggling to prepare data for GPU/s and GPU/s  are spending a lot of time waiting for the data or is it perhaps vice versa and the model is too big,  so batches are waiting to come into GPU for too long. It could also be the problem to utilize all GPUs correctly using distributed parallel training and so on.