You could just slice the state and take the appropriate slice of data. This has the drawback that PyTorch cannot fuse the matmuls as nicely anymore, but it should not be unreasonable.
But my understanding was that you preferred masking…