How are you performing model parallelism? Are you using PyTorch RPC? A simple reproducible script would help a lot in understanding the root cause of the issue.
Since your model works with 1 GPU but not 2, I am just guessing it may have something to do with cuda synchronization and this is giving garbage values, but please post an example of the code, your PyTorch version, and any frameworks you are using.