Besides the possible data loading bottleneck, small architectures also might result in a low GPU utilization.
Do you see some peak utilization and then a small pause or is the utilization approx. the whole time at 30%?
Could you explain your model architecture a bit?