Missing activation Function
I think the sigmoid / softmax activation function is missing here

https://github.com/huggingface/transformers/blob/31ec2cb2badfbdd4c1ac9c6c9b8a74e974984206/src/transformers/models/roberta/modeling_tf_roberta.py#L1299
Hey @RodSernaPerez 👋  The `sigmoid` activation function was added to our list of TF activation functions, it should be operational now (you will have to install `transformers>=4.19.0.dev0` or pull from `main`). 

Feel free to reopen the issue if you run into problems :)