https://pytorch.org/blog/optimizing-cuda-rnn-with-torchscript/ 753 is a good tutorial. In particular, you can copy & paste from this file: https://github.com/pytorch/pytorch/blob/cbcb2b5ad767622cf5ec04263018609bde3c974a/benchmarks/fastrnns/custom_lstms.py#L60-L83 946 that has a layernorm lstm implementation.