Hi,
unfold and fold should be very fast as they only play with stride in general. Are you sure they are responsible for the slowdown.
If you compare with a general convolution algorithm, depending on the input, it is not always the most efficient to do the unfold, mm, fold.