Thanks for the suggestion. I was using nn.Linear to illustrate my problem. In reality, my model consists of two deep convolutions networks.
If it is simply a nn.Linear, your suggestions does work. But how about if it is a deep convolutions networks?