I found the solution! The mistake I made is that I use two element to indicate the binary label in pytorch,
the correct format for positive label is 1 and negative label is 0, but I wrote (1,0) and (0,1)