Finding the right keras loss and metric for SQuAD
## ❓ Questions & Help

<!-- A clear and concise description of the question. -->
I am trying to build a simple example with BERT for QA (on SQuAD). The goal is getting it similarly as simple as [the GLUE example from the repository](https://github.com/huggingface/transformers#quick-tour-tf-20-training-and-pytorch-interoperability). 

The problem I am facing is finding the accurate loss function and metric. According to the [documentation](https://huggingface.co/transformers/model_doc/bert.html#tfbertforquestionanswering), TFBertForQuestionAnswering will return the logits for a span_start and span_end prediction. 
Since we want one single loss for both predictions, one could use the reduced sum of the both categorical crossentropies of the span predictions.

Is that a sensible way to do it? 
Is there another, better way? 
Can we "stack" losses in keras or is it just not possible? 

I am thankful for any help.

For reference: My current state can be found in [this colab notebook](https://colab.research.google.com/drive/1xDpV0z3432mnqdvDC40kMi-KQWxiKPQK)
 

In case @jplu has an insight on this?<|||||>Hi @jwallat,

You might have two solutions to solve your issue:

* Implement your own loss function that you can give to the compile method (see the official Tensorflow [doc](https://www.tensorflow.org/api_docs/python/tf/keras/Model?version=stable#compile))
* Implement a custom training loop such as the `train` function in the [NER example](https://github.com/huggingface/transformers/blob/master/examples/run_tf_ner.py#L154)