You have used 4 for-loops in your convolution algorithm. I guess it negates any performance you can achieve with GPU utilization. This fastai 3 tutorial might be helpful I guess. Notice, how when they reduce the loops the speed increases dramatically.