Thank you for help. The problem I had was in opencv side, I had to change cols to rows.
Also, if you have the time. How good is your performance? I am using stream dataloader that creates opencv mat images on the fly, however im getting cpu usage of about 60% and GPU of about 50% which seems way too low.