This one uses 80% of my tiny GPU.
Also, I just solved a few inconsistencies in my code. the o/p. class labels and stuff. It turns out I actually had 21 classes instead of 20. I just solved this, restarted my machine and now it somehow works!!
Thanks a lot @ptrblck. You rock!