In the batch first mode (which is what you have), the output has this meaning:
batch * seq_len * vector size
So with a -1 in the middle dimension, you are extracting the very last output of time series output sequence of the RNN. This final output represents the summary of the entire sequence and that is what you want for a many to one scenario. The other :'s just means you want that last output for all batch and all input vectors.