Hello,
from both the text you quoted and the code, I’d say likely option 2. This is my understanding of layers.
If you believe that, after disabling cudnn, this line is relevant, it might also give a hint.

  
      github.com
  
  
    pytorch/pytorch/blob/master/torch/nn/_functions/rnn.py#L91 85

    for j, inner in enumerate(inners):
        l = i * num_directions + j


        hy, output = inner(input, hidden[l], weight[l])
        next_hidden.append(hy)
        all_output.append(output)


    input = torch.cat(all_output, input.dim() - 1)


    if dropout != 0 and i < num_layers - 1:
        input = F.dropout(input, p=dropout, training=train, inplace=False)


if lstm:
    next_h, next_c = zip(*next_hidden)
    next_hidden = (
        torch.cat(next_h, 0).view(total_layers, *next_h[0].size()),
        torch.cat(next_c, 0).view(total_layers, *next_c[0].size())
    )
else:
    next_hidden = torch.cat(next_hidden, 0).view(
        total_layers, *next_hidden[0].size())



  
  
    
    
  
  


My understanding of this code is that it does a single timestep (your horizontal axis) for many layers.
Best regards
Thomas