Just a quick update: Thanks to Ayrton San Joaquin the SELU gain we found here is now in PyTorch 1.8: Add SELU Activation to calculate_gain by ajsanjoaquin · Pull Request #50664 · pytorch/pytorch · GitHub 9