If you cannot write everything with tensors, you will have to write a custom autograd.Function and write the backward pass yourself. It is usually tricky to do.
You can post here if you need any help.