Isn't it too few activations?
Hello, I have a question. In BertLayer part of model we see, that in BertAttention module we do attention (nonlinear action) and selfOutput (linear transformation as it is dense + BN). Then we do BertIntermediate, starting with linear transformation(which means, we have dense, BN, dense transformations going one by one). Why don't we have nonlinear activation between BertAttention and BertIntermediate in BertLayer?
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
