Seems like that there are some problems or pitfalls in Conv2d on GPU. When I used in_channels = 1, out_channels = 200, kernel height = 3, kernel width = 300 (original setting) then backward was very slow. I changed it to in channels = 300, out channels = 200, kernel height = 1, kernel width = 3 then model works well. (I got 9500/sec on GTX 1080 8) )
This seems like GPU-specific problem because no problem occurred when ran on CPU.