I’m getting started in PyTorch and have a few years experience with Tensorflow v1. I’m a bit confused about how RNNs work in PyTorch.
It seems to me that the provided RNNs in ‘nn’ are all C implementations and I can’t seem to find an equivalent to Tensorflow’s ‘scan’ or ‘dynamic_rnn’ function. Furthermore, all custom implementations of RNNs in PyTorch seem to work using Python for-loops. Wouldn’t this result in multiple calls to the GPU, which slows everything down?
Second: I am used to dealing with minibatches containing variable length sequences in Tensorflow by providing the length of each sequence to the RNN function. This has the advantage (over using an explicit pad token) that you do not need to create a useless entry in the vocabulary just for a pad token. The padded data can be any existing token which will be ignored because it lies outside of the declared length. But it seems that PyTorch assumes a pad token the same way that Keras does. Am I understanding this right? Do I have to reserve an entry in my embedding matrix for pad tokens?
Thanks!