This is expected, as unfold would explicitly create the matrix so that you are able to perform the matmul afterwards. Some libraries such as cudnn ship with tuned algorithms (for common conv layer setups) in order to speed up the operation.