It’s an interesting approach 
In fact, the convolutions seems to be faster. Here are the timings for a GTX1070 and an old i7 CPU:
GPU, 2d, 500 repetitions:
Conv2d: 0.14418 sec
Linear: 1.52989 sec

CPU, 2d, 500 repetitions:
Conv2d: 6.34300 sec
Linear: 11.29293 sec

--------------------------

GPU, 1d, 500 repetitions:
Conv1d: 0.147367 sec
Linear: 1.52954 sec

CPU, 1d, 500 repetitions:
Conv1d: 6.43247 sec
Linear: 12.29751 sec

Note that the difference between the 1D and 2D case should be minimal, as I’ve used h, w = 128, 128 for 2D and h = 128*128 for 1D.
Maybe the workloads are unrealistic?