Sorry I made a mistake in my comment above, the .cuda() is not needed.
torch.zeros(B, S)

is equivalent to
torch.Tensor(B, S).zero_()

By default torch.Tensor == torch.FloatTensor so no it won’t be on gpu.