How are you checking the “updated channels”? Assuming you are using an nn.Conv2d layer as the output layer, each class channel would be created by a separate filter, which would use all input channels by default. So depending what exactly you are checking the updates might be expected.