jpainam:

My understanding of pooling (2D or 3D) is that it only works on the spatial dimension.


No, that’s not the case and is the difference between the 2D and 3D layers.
While the 2D layers apply the kernel on the H and W dimension for an input of [N, C, H, W], 3D layers will use a 3D kernel applied on D, H, W for an input of [N, C, D, H, W].