@ptrblck alpha is parameter of a custom activation function, and is different each time I use it in multiple layers like batchnorm. How do I pass it to optimizer in my training routine? There should be away to do it automatically like linear or batchnorm  layer where we do loss.backward followed by optimizer.step() and all parameters gets updated?
My next query is:
I want to get rid of weight variable and use alpha directly. Since alpha is nn.Parameter I can’t assign it with output of get_alpha() function. masked_scatter_ gives me CUDA errors.
Thanks