BERT includes a linear + tanh layer as the pooler. I recently wrote a very compact implementation of BERT Base 2 that shows what is going on. L354 you have the pooler, below is the BERT model. The last_hidden_state is the output of the blocks, you can set model.pooler to torch.nn.Identity() to get these, as shown in the test 3 which shows how to import BERT from the HF transformer library into this simple transformer implementation. (I also have a DistillBert version but not yet pushed to github. DistilBert drops the pooler. Interestingly, the models are coded quite differently internally, even though they use the same computation…)
Best regards
Thomas