Thank you for your reply @wanchaol , the bug has been solved.
The reason is that my model will not use postnet to compute loss in some cases (there is a if condition in the code), so the back-propagation will fail.