[Benchmark] QUAERO French Medical Corpus for Named Entity Recognition
# 🖥 Benchmarking `transformers`

## Benchmark

I am trying the Transformers library out on [QUAERO French Medical Corpus](https://quaerofrenchmed.limsi.fr/) NER-dataset (consisting of Medline titles and EMEA documents). Using Camembert-base 'out of the box' with default hyperparameters, I get an F1 measure of 85% on the EMEA dataset and 64% on Medline, while [one of the few papers](http://ceur-ws.org/Vol-1391/158-CR.pdf) I found that did an experiment on the same dataset reported F1 of 70% and 52%, respectively, using a classic CRF. 

Since extensive grid search for hyperparameter optimization is computationally expensive even with a GPU, and given that I am relatively new to the field, I was wondering how to actually go about further optimizing the current model that I have. Is it even worth doing a lot of hyperparameter optimization or is it common that out of the box transformer models already do a decent job?

In particular, I'm not so sure which of the following things are worthwhile trying out;
- Making the input for the BERT model longer than 1 sentence
e.g. 3 input sentences [CLS] sentence1 sentence2 sentence3 [SEP]
- Extending the vocab of the tokenizer by adding tokens related to the medical domain? Not sure 
if this even makes sense doing..
- Which hyperparameters do I focus on the most? 
I assume epochs and batch size are the most important, but which others are worth trying out as well?
- Other suggestions to improve this model?

If you need more information regarding the setup I will be happy to provide it, thanks!


This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
