Hi Rui An!



 ruian1:

Could you also point me to the block where “log-sum-exp” is applied to ? I traced it back to functional.py but still can’t figure out how it is done…


I don’t have the binary_cross_entropy_with_logits code
in front of me, so I can’t give you the specifics.
The issue is that floating-point error can get amplified when you
compute log of an expression containing exp.  sigmoid has
the exp, and cross-entropy has the log, so you can run into
this problem when using sigmoid as input to cross-entropy.
Dealing with this issue is the main reason that
binary_cross_entropy_with_logits exists.
See, for example, the comments about “log1p” in the Wikipedia
article about logarithm 2.
(I was speaking loosely when I mentioned the related
“log-sum-exp-trick.”  This is more directly relevant to computing
softmax, but is basically another facet of the same issue.  For
more on this, see, for example, Wikipedia’s LogSumExp 10)
Best.
K. Frank