Solved by zeke in post #5 
              
              
                I monitored GPU usage via nvidia-smi. I also increased the network’s size. It turns out that the network was too small to be fully utilized by GPU. Increasing it’s size increased GPU usage.