No as seen using your code snippet:
model = Net()
loss_func = nn.MSELoss()
optimizer = torch.optim.SGD(model.parameters(),lr=1.)

model.fc3.weight = nn.Parameter(model.fc3.weight/torch.max(model.fc3.weight))

out = model(torch.randn(1, 2))
out.mean().backward()

print(model.fc3.weight)
> Parameter containing:
tensor([[ 0.9226, -0.6592, -0.6971,  0.8911],
        [-0.8582,  0.9190,  0.2713,  1.0000]], requires_grad=True)

optimizer.step()
print(model.fc3.weight)
> Parameter containing:
tensor([[ 0.9226, -0.6592, -0.6971,  0.8911],
        [-0.8582,  0.9190,  0.2713,  1.0000]], requires_grad=True)

print(model.fc3.weight.grad)
> tensor([[ 0.0622, -0.0140, -0.0225, -0.0629],
        [ 0.0505, -0.0114, -0.0183, -0.0511]])

optimizer.zero_grad()
print(model.fc3.weight.grad)
> tensor([[ 0.0622, -0.0140, -0.0225, -0.0629],
        [ 0.0505, -0.0114, -0.0183, -0.0511]])

You would have to add the new parameter to the optimizer again.