You pass (some) parameters to the optimizer, which are updated using their gradients (stored in the .grad attribute of each parameter). If you don’t pass certain parameters to an optimizer, they won’t be updated in any way. Since softmax does not contain any parameters, your current idea won’t work unfortunately.


If a layer is not used during the forward pass, it won’t get any gradients in the backward pass. You could therefore use an if condition and use different layers in the forward pass. Only the selected (and used) layer will be included in the backward pass. This will add some overhead, but you could transfer all unused layers back to the CPU to free the GPU memory for the current one.


I would generally “Go for it!”, especially after reading your first sentence 



 bonzogondo:

I have a weird question about some crazy idea I want to materialize using PyTorch.


However, even if you skip the idea of using different softmax layers (due to the missing parameters) and instead would use e.g. different linear layers, I’m not sure how you would chose between them in the test case, i.e. if you are dealing with new samples, which are not coming from the three predefined Datasets.