It could be slower, but not that much.
================================================================================
Common FC layer Test
--------------------------------------------------------------------------------
Construct and Forward Time:
-------------  ---------------  ---------------  ---------------  ---------------  ---------------
Name                  CPU time        CUDA time            Calls        CPU total       CUDA total
-------------  ---------------  ---------------  ---------------  ---------------  ---------------
unsqueeze             13.605us          0.000us                2         27.210us          0.000us
mm                  5156.834us          0.000us                2      10313.668us          0.000us
squeeze_               5.965us          0.000us                2         11.929us          0.000us

Elapsed 61.172 ms
--------------------------------------------------------------------------------
Backward Time:
-------------------------------------  ---------------  ---------------  ---------------  ---------------  ---------------
Name                                          CPU time        CUDA time            Calls        CPU total       CUDA total
-------------------------------------  ---------------  ---------------  ---------------  ---------------  ---------------
N5torch8autograd9GraphRootE                    5.604us          0.000us                1          5.604us          0.000us
SqueezeBackward1                               3.305us          0.000us                2          6.610us          0.000us
unsqueeze                                      2.368us          0.000us                2          4.736us          0.000us
MmBackward                                  2872.008us          0.000us                2       5744.016us          0.000us
t                                              3.864us          0.000us                3         11.593us          0.000us
mm                                          1905.354us          0.000us                3       5716.063us          0.000us
UnsqueezeBackward                              2.018us          0.000us                1          2.018us          0.000us
squeeze                                        1.411us          0.000us                1          1.411us          0.000us
N5torch8autograd14AccumulateGradE           2083.309us          0.000us                2       4166.618us          0.000us
N5torch8autograd5CloneE                     2078.855us          0.000us                2       4157.710us          0.000us

Elapsed 28.548 ms
================================================================================
Stupid FC layer Test
--------------------------------------------------------------------------------
Construct and Forward Time:
-------------  ---------------  ---------------  ---------------  ---------------  ---------------
Name                  CPU time        CUDA time            Calls        CPU total       CUDA total
-------------  ---------------  ---------------  ---------------  ---------------  ---------------
narrow                 0.752us          0.000us             3000       2255.612us          0.000us
unsqueeze              0.717us          0.000us             3001       2150.282us          0.000us
mm                    24.225us          0.000us             3001      72700.511us          0.000us
squeeze_               0.579us          0.000us             3001       1736.924us          0.000us
SetItem                8.354us          0.000us             3000      25061.452us          0.000us

Elapsed 232.678 ms
--------------------------------------------------------------------------------
Backward Time:
-------------------------------------  ---------------  ---------------  ---------------  ---------------  ---------------
Name                                          CPU time        CUDA time            Calls        CPU total       CUDA total
-------------------------------------  ---------------  ---------------  ---------------  ---------------  ---------------
N5torch8autograd9GraphRootE                    4.172us          0.000us                1          4.172us          0.000us
SqueezeBackward1                               3.720us          0.000us             3001      11162.951us          0.000us
unsqueeze                                      2.465us          0.000us             3001       7398.137us          0.000us
MmBackward                                    21.510us          0.000us             3001      64551.329us          0.000us
t                                              1.542us          0.000us             3002       4630.286us          0.000us
mm                                            16.302us          0.000us             3002      48937.424us          0.000us
UnsqueezeBackward                              1.444us          0.000us                1          1.444us          0.000us
squeeze                                        0.937us          0.000us                1          0.937us          0.000us
N5torch8autograd14AccumulateGradE           2602.311us          0.000us                2       5204.621us          0.000us
N5torch8autograd5CloneE                     2598.121us          0.000us                2       5196.242us          0.000us
SetItemBackward                               90.268us          0.000us             3000     270804.749us          0.000us
clone                                          9.292us          0.000us             3000      27875.561us          0.000us
SetItem                                       28.317us          0.000us             3000      84951.591us          0.000us
Index                                         11.494us          0.000us             3000      34483.173us          0.000us
view                                           5.437us          0.000us             3000      16311.041us          0.000us
NarrowBackward                              2183.574us          0.000us             3000    6550722.268us          0.000us
_unnarrow                                   2181.987us          0.000us             3000    6545962.471us          0.000us
N5torch8autograd3AddE                       5623.963us          0.000us             2999   16866264.974us          0.000us

Elapsed 24832.020 ms

It’s profiler’s result. It doesn’t make sense that a stupid fc layer([3000] * [3000, 1] matmul operations 3000 times) will need 25s for backward operation.
It shows that most time consumed by NarrowBackward/unnarrow and autograd add operations.
I have no idea how does it implement inside. In my opinion, narrowbackward and unnarrow are mostly index computing operations, and autograd.add are mostly element-wise add operations, and autograd.add are much much slower than common element-wise addition operations.