Oh, I might have misunderstood your use case and though you would like to keep one of the observations only. I.e. either zero out x[:, :, 0, :] completely or x[:, :, 1, :].
Your current approach would randomly zero out values in the observations dimension, wouldn’t it?