Thank you @ptrblck. My bad
Here is the updated one: 
new683×665 77.8 KB

Yes, I agree with you as well. So, I suspected that my sampling of data for the network should be an issue. To not miss anything, I made the data loader to fetch similarly sampled data just as in Caffe to the network and I found the same kind of poor convergence as before without modifying the data loader