Great !
Choosing lr=0.1 does the trick:
Screen Shot 2020-09-15 at 17.05.34520×1668 178 KB Screen Shot 2020-09-15 at 17.05.39772×1086 63.4 KB
Still I fail to understand what the problem is. After all I determined the parameter in a nn.Module (LearnableAugs) and then passed it to be processed in an internal nn.Module that doesn’t have those registered as parameters. Nevertheless, I expect the gradients to flow properly as happening for all the other transforms I learn (e.g. rotation, shear, etc.). It’s a mystery to me.