Sounds like an interesting project.
It seems you are trying to learn the rotation matrix, but apparently you get some invalid values in the last row.
Would it make sense to split the rotation and translation part of your transformation and try to learn the rotation then?
It wouldn’t be that performant, but maybe it’ll help the model to learn.