Metrics are coupled to the run_glue.py tasks.
The metrics used for evaluating a run_glue task is coupled to the task itself (i.e. regression or classification). We wrote a `TrinarySentimentProcessor` which grabs a positive, neutral, or negative sentiment from a text, but we found that the simple accuracy measure was not the right one. We wanted to use log loss (cross entropy) and so we added an log loss function to the metrics (`__init__.py`) package and added another elif to the chain.

Why is the metric so coupled to the task itself? What if you wanted to use a different metric for any particular task? I understand that `run_glue.py` is an "example" but we've been building upon the architecture to reduce the workload of training for tasks outside of GLUE and SQUAD.  Maybe you could add a flag to the `run_glue.py` file like `--metric=cross-entropy`. Any thoughts? Are we just misusing the library by extending `glue.py`?
As you say yourself, the core of the library is to provide you with models (and recently also pipelines and in the future even more exciting things) to build your own projects. The examples show you some implementations that are in themselves usable but that are by no means meant to be exhaustive for all problems. As you indicate yourself, you are invited to adapt these examples to your own use case.

If you feel that your changes are useful for the whole community, then feel free to request a PR!<|||||>This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
