Then perhaps 4d input is appropriate:
(batch_size=1, channels=3 [xyz], objects=m, timesteps=k)
note that batch_size=m is initially incorrect if objects are not independent (unless later layers add autoregressive dependencies)
With above 4d input, you can perhaps do two separate causal convolutions 1x3 and 3x1. That’s not something I’ve ever done, so not sure.