You are right, that most activation functions don’t have parameters, so that you don’t have to initialize them.
However, you use the gain corresponding to the non-linearity you’ve used in your model.
E.g. for a model like conv -> relu -> conv -> relu, you could use:
nn.init.xavier_uniform_(m, gain=nn.init.calculate_gain('relu'))

to initialize the weights of the conv layers.