So since the second “column” has steady values, your model does not learn anything?
Do you think my approach is applicable at all in your use case, e.g. do you need to backprop through the 4 activations with the additional Cond2d layers?
If so, I think the better way would be to re-implement the model and output the desired activations. This would make sure that the model is being trained.