Under the hood, not exactly. There should be some more optimizations to parallel-process (something like this 3)  taking advantage of GPUs for class of RNN Models. These are implemented in the CPP backend (https://github.com/pytorch/pytorch/blob/master/torch/nn/_VF.py 9) .
But you should be able to approximate the behaviour using composition of nn.Linear layers, I think. GRU and LSTM will serve as a substitute which could be adapted for your use case - but your query seemed like you wanted more flexibility.