THNN source code is little bit confusing because there are too many macro references, sometimes nested. But as I see, it first computes bias via GEMM, and apply im2col to the filter and input, with another GEMM following. As GEMMs are supposed to take same amount of computation, I think the bottleneck could be im2col procedure. So It might be hard to optimize.