I couldn’t find any direct solutions and ended up permuting the dimensions. For anyone that ends up here, this is how I did the downsampling:
F.interpolate(input.permute(1, 2, 0), scale_factor=..., mode=...).permute(2, 0, 1)

The output can now be fed into modules like GRU.