It is supported for most things, one notable exception are the cudnn recurrent modules (like nn.LSTM). You can use torch.autograd.grad in your forward and use .backward just as you usually do in PyTorch.
Best regards
Thomas