Thank you for your reply. The problem has not yet solved.
If I remove the clip_norm in the training step, the gradient will explode after some batches. The training process looks like okay before explosion. No norm operation is used in my case. All input tensors and ground truth tensors are normalized into [0,1]. L2 loss and weight_decay are used. I have no idea which detail should I concentrate on.