I was able to track down my errors by carrying everything out in “world” coordinates and then creating the affine grid from them.
I had to do a lot of “visual” debugging with https://github.com/K3D-tools/K3D-jupyter 12 which I used for rendering my point clouds and input/output data (really a great tool for that).