So what I did was to implement autograd.Functions that use the regular forward but implement the LRP rules for backward. Then I mirrored all nn.Modules my network uses. Worked for LSTMs (ULMFiT) and ResNets and some others for me.
Note that “unconstrained” LRP for ResNets does have continuity issues with the residual connections, here is a discussion involving some coauthors of some papers 161.
Best regards
Thomas