I am facing same problem when working ImageNet data, things are alright when I work with other datasets like Pascal VOC. Volatile gpu util is also showing 0% most of the time. Any solution for this problem?