Could you use the output directly to test the specific block via output.mean().backward() in both frameworks?
I would still recommend to scale down the issue and start with small unit tests, as you are currently testing multiple modules at once.