Thanks @saareliad for looking that up. I was going to do that, glad you did…
So far for “scale by parameter size = True” – in my experiments

This does works.
The model converges much more slowly (for fine-tuning, on 8 GPU Volta) – and from what I saw to a worse number.

But yes, it’s a parameter that does work. Maybe need to increase the LR or train for much longer with that on.