TAPAS tanh activation on the pooling layer
I noticed the following in the TAPAS pooling layer:

https://github.com/huggingface/transformers/blob/d83b0e0c079f0826d186270a86622ff5f1efd9c1/src/transformers/models/tapas/modeling_tapas.py#L696-L709

I'm curious about the use of `nn.Tanh()`. I wasn't able to find more information about that activation in [the paper](https://arxiv.org/abs/2004.02349). Is it possible to know where it comes from? Thanks!
Hi,

The TAPAS authors borrowed this from the original BERT paper, which decided to apply a tanh layer. 

The BERT author explains why he did that [here](https://github.com/google-research/bert/issues/43).<|||||>Ah thanks, you are right. They indeed use tanh in the code: https://github.com/google-research/tapas/blob/f3d9f068e6eedb252883049b582516a1294ff951/tapas/models/bert/modeling.py#L269-L277

Wish it was mentioned in the appendix of the TAPAS paper 🤷 Thanks for clarifying!