I also calculated the cosine similarity of the activations in each of the layers:
conv_stem.0.stats tensor(0.9992)
conv_stem.1.stats tensor(0.9992)
blocks.0.0.conv_dw.0.stats tensor(0.9882)
blocks.0.0.conv_dw.1.stats tensor(0.9882)
blocks.0.0.conv_pw.stats tensor(0.9376)
blocks.1.0.conv_pw.0.stats tensor(0.9126)
blocks.1.0.conv_pw.1.stats tensor(0.9569)
blocks.1.0.conv_dw.0.stats tensor(0.9549)
blocks.1.0.conv_dw.1.stats tensor(0.9677)
blocks.1.0.conv_pwl.stats tensor(0.8856)
blocks.1.1.conv_pw.0.stats tensor(0.9488)
blocks.1.1.conv_pw.1.stats tensor(0.9625)
blocks.1.1.conv_dw.0.stats tensor(0.9364)
blocks.1.1.conv_dw.1.stats tensor(0.9385)
blocks.1.1.conv_pwl.stats tensor(0.8623)
blocks.2.0.conv_pw.0.stats tensor(0.9364)
blocks.2.0.conv_pw.1.stats tensor(0.9518)
blocks.2.0.conv_dw.0.stats tensor(0.9711)
blocks.2.0.conv_dw.1.stats tensor(0.9819)
blocks.2.0.conv_pwl.stats tensor(0.8685)
blocks.2.1.conv_pw.0.stats tensor(0.9585)
blocks.2.1.conv_pw.1.stats tensor(0.9671)
blocks.2.1.conv_dw.0.stats tensor(0.9565)
blocks.2.1.conv_dw.1.stats tensor(0.9647)
blocks.2.1.conv_pwl.stats tensor(0.7922)
blocks.3.0.conv_pw.0.stats tensor(0.9168)
blocks.3.0.conv_pw.1.stats tensor(0.9344)
blocks.3.0.conv_dw.0.stats tensor(0.9773)
blocks.3.0.conv_dw.1.stats tensor(0.9831)
blocks.3.0.conv_pwl.stats tensor(0.8967)
blocks.3.1.conv_pw.0.stats tensor(0.9597)
blocks.3.1.conv_pw.1.stats tensor(0.9683)
blocks.3.1.conv_dw.0.stats tensor(0.9532)
blocks.3.1.conv_dw.1.stats tensor(0.8986)
blocks.3.1.conv_pwl.stats tensor(0.8574)
blocks.3.2.conv_pw.0.stats tensor(0.9549)
blocks.3.2.conv_pw.1.stats tensor(0.9664)
blocks.3.2.conv_dw.0.stats tensor(0.9599)
blocks.3.2.conv_dw.1.stats tensor(0.8697)
blocks.3.2.conv_pwl.stats tensor(0.7916)
blocks.4.0.conv_pw.0.stats tensor(0.9387)
blocks.4.0.conv_pw.1.stats tensor(0.9521)
blocks.4.0.conv_dw.0.stats tensor(0.9650)
blocks.4.0.conv_dw.1.stats tensor(0.9685)
blocks.4.0.conv_pwl.stats tensor(0.8268)
blocks.4.1.conv_pw.0.stats tensor(0.9460)
blocks.4.1.conv_pw.1.stats tensor(0.9414)
blocks.4.1.conv_dw.0.stats tensor(0.9698)
blocks.4.1.conv_dw.1.stats tensor(0.9566)
blocks.4.1.conv_pwl.stats tensor(0.7595)
blocks.4.2.conv_pw.0.stats tensor(0.9490)
blocks.4.2.conv_pw.1.stats tensor(0.9423)
blocks.4.2.conv_dw.0.stats tensor(0.9809)
blocks.4.2.conv_dw.1.stats tensor(0.9683)
blocks.4.2.conv_pwl.stats tensor(0.7715)
blocks.5.0.conv_pw.0.stats tensor(0.9567)
blocks.5.0.conv_pw.1.stats tensor(0.9359)
blocks.5.0.conv_dw.0.stats tensor(0.9930)
blocks.5.0.conv_dw.1.stats tensor(0.9949)
blocks.5.0.conv_pwl.stats tensor(0.9064)
blocks.5.1.conv_pw.0.stats tensor(0.9649)
blocks.5.1.conv_pw.1.stats tensor(0.9595)
blocks.5.1.conv_dw.0.stats tensor(0.9741)
blocks.5.1.conv_dw.1.stats tensor(0.9583)
blocks.5.1.conv_pwl.stats tensor(0.8771)
blocks.5.2.conv_pw.0.stats tensor(0.9700)
blocks.5.2.conv_pw.1.stats tensor(0.9648)
blocks.5.2.conv_dw.0.stats tensor(0.9810)
blocks.5.2.conv_dw.1.stats tensor(0.9709)
blocks.5.2.conv_pwl.stats tensor(0.8566)
blocks.5.3.conv_pw.0.stats tensor(0.9687)
blocks.5.3.conv_pw.1.stats tensor(0.9679)
blocks.5.3.conv_dw.0.stats tensor(0.9842)
blocks.5.3.conv_dw.1.stats tensor(0.9755)
blocks.5.3.conv_pwl.stats tensor(0.8879)
blocks.6.0.conv_pw.0.stats tensor(0.9637)
blocks.6.0.conv_pw.1.stats tensor(0.9655)
blocks.6.0.conv_dw.0.stats tensor(0.9802)
blocks.6.0.conv_dw.1.stats tensor(0.9842)
blocks.6.0.conv_pwl.stats tensor(0.8535)
conv_head.0.stats tensor(0.6853)
conv_head.1.stats tensor(0.6853)
quant.stats tensor(0.9998)
classifier.stats tensor(0.7695)

it seems as if the conv_pwl layers are mostly different, along with the conv_head and classifier.
What might cause that?