If your new function is differentiable then just write it as a python function. If it has parameters, you can use nn.Module and you will need to implement the init and the forward for your function.
If it is not differentiable, you will have to define the backward operation you want using the informations here 66.