Hello all,
i’m trying to freeze all parameters of my model. “param.required grad = False” is very simple and powerful way that most of developer accept, but i failed to confirm the effect of that.
For the simplest test to check whether freezing is works, first i initailze model and assign param.requried grad as False. i run train function for the model, and visualize very few param from the model.
I expect visualized params won’t changed at all because i freeze the parameters, but the param keep changing when the loop is going on one batch to another. The belows are very psuedocode for my work, and what terminal shows.
anybody has idea??
Thank you.
model = init.model()
for param in model.parameters():
    param.required_grad = False

optimizer = torch.optim.RSMprop(model.parameters(), lr=args.lr, momentum=args.momentum,weight_decay = args.weight.decay)

model.train()
for i (inputs, target, meta) in enumerate(train_loader):
    outputs=model(inputs)
    loss = criterion(inputs,target)
    optimizer.zero_grad()
    loss.backward()
    optimizer.step()

    ## visualize very few params
    for param in model.parameters():
        print(param[0])
        break

At the terminal shows like
Processing |#######                         | (509/2225) Data: 0.052407s | Batch: 1.401s | Total: 0:09:21 | ETA: 0:32:52 | Loss: 0.0322 | Acc:  0.0236Variable containing:
(0 ,.,.) =
1.00000e-02 *
-5.1111  4.8789  7.2060  4.0981  1.5398 -1.9935  2.5382
-6.8706 -1.9746  5.5094 -1.3251  0.4653  3.0574 -0.6753
5.6140 -7.1054  6.3074  6.6501 -1.3995  6.1489  1.1402
0.1733  3.8403  2.5650 -3.3059 -1.5468  6.1374 -2.1767
8.3077 -0.2914 -7.0128  1.2817  7.1735 -6.6335 -1.9335
-1.6674 -3.0738  2.7778 -3.6561 -0.6739  3.6340  3.9714
7.2121 -0.4230  1.4884 -6.0607  2.8108  8.2189  3.3358
Processing |#######                         | (510/2225) Data: 0.015237s | Batch: 1.283s | Total: 0:09:22 | ETA: 0:33:58 | Loss: 0.0321 | Acc:  0.0236Variable containing:
(0 ,.,.) =
1.00000e-02 *
-5.1005  4.8870  7.2142  4.1049  1.5466 -1.9869  2.5436
-6.8591 -1.9656  5.5182 -1.3171  0.4710  3.0624 -0.6712
5.6252 -7.0960  6.3153  6.6564 -1.3954  6.1527  1.1435
0.1828  3.8488  2.5727 -3.2997 -1.5428  6.1408 -2.1743
8.3194 -0.2824 -7.0051  1.2889  7.1788 -6.6295 -1.9307
-1.6547 -3.0628  2.7874 -3.6472 -0.6671  3.6397  3.9762
7.2225 -0.4128  1.4972 -6.0499  2.8183  8.2252  3.3417