If I understood the problem correctly, you first convert every 16 frames from a video to a 512-D vector which means if your video has, say, n frames you’ll get n/16 of 512-D vectors. Your sequence length is n/16. But you said its 16. Are all videos 256 frames long?
So if you know your ‘n/16’, use tensor.view(batch_size, n/16, 512). I’m not sure if I answered your question. It will be better if you mention the paper you’re implementing.