If you would like to create a new layer, you could

create a new nn.Module writing your forward method, if you stick to PyTorch methods
create a torch.autograd.Function with a custom forward and backward method, if you need to leave PyTorch or would like to implement it manually

Have a look at the Extending PyTorch docs 223.