<meta charset="utf-8" /><meta http-equiv="content-security-policy" content=""><meta name="hf:doc:metadata" content="{&quot;local&quot;:&quot;choosing-a-metric-for-your-task&quot;,&quot;sections&quot;:[{&quot;local&quot;:&quot;categories-of-metrics&quot;,&quot;sections&quot;:[{&quot;local&quot;:&quot;generic-metrics&quot;,&quot;title&quot;:&quot;Generic metrics&quot;},{&quot;local&quot;:&quot;taskspecific-metrics&quot;,&quot;title&quot;:&quot;Task-specific metrics&quot;},{&quot;local&quot;:&quot;datasetspecific-metrics&quot;,&quot;title&quot;:&quot;Dataset-specific metrics&quot;}],&quot;title&quot;:&quot;Categories of metrics&quot;}],&quot;title&quot;:&quot;Choosing a metric for your task&quot;}" data-svelte="svelte-1phssyn">
	<link rel="modulepreload" href="/docs/evaluate/pr_36/en/_app/assets/pages/__layout.svelte-hf-doc-builder.css">
	<link rel="modulepreload" href="/docs/evaluate/pr_36/en/_app/start-hf-doc-builder.js">
	<link rel="modulepreload" href="/docs/evaluate/pr_36/en/_app/chunks/vendor-hf-doc-builder.js">
	<link rel="modulepreload" href="/docs/evaluate/pr_36/en/_app/chunks/paths-hf-doc-builder.js">
	<link rel="modulepreload" href="/docs/evaluate/pr_36/en/_app/pages/__layout.svelte-hf-doc-builder.js">
	<link rel="modulepreload" href="/docs/evaluate/pr_36/en/_app/pages/choosing_a_metric.mdx-hf-doc-builder.js">
	<link rel="modulepreload" href="/docs/evaluate/pr_36/en/_app/chunks/Tip-hf-doc-builder.js">
	<link rel="modulepreload" href="/docs/evaluate/pr_36/en/_app/chunks/IconCopyLink-hf-doc-builder.js">
	<link rel="modulepreload" href="/docs/evaluate/pr_36/en/_app/chunks/CodeBlock-hf-doc-builder.js"> 





<h1 class="relative group"><a id="choosing-a-metric-for-your-task" class="header-link block pr-1.5 text-lg no-hover:hidden with-hover:absolute with-hover:p-1.5 with-hover:opacity-0 with-hover:group-hover:opacity-100 with-hover:right-full" href="#choosing-a-metric-for-your-task"><span><svg class="" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" aria-hidden="true" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 256 256"><path d="M167.594 88.393a8.001 8.001 0 0 1 0 11.314l-67.882 67.882a8 8 0 1 1-11.314-11.315l67.882-67.881a8.003 8.003 0 0 1 11.314 0zm-28.287 84.86l-28.284 28.284a40 40 0 0 1-56.567-56.567l28.284-28.284a8 8 0 0 0-11.315-11.315l-28.284 28.284a56 56 0 0 0 79.196 79.197l28.285-28.285a8 8 0 1 0-11.315-11.314zM212.852 43.14a56.002 56.002 0 0 0-79.196 0l-28.284 28.284a8 8 0 1 0 11.314 11.314l28.284-28.284a40 40 0 0 1 56.568 56.567l-28.285 28.285a8 8 0 0 0 11.315 11.314l28.284-28.284a56.065 56.065 0 0 0 0-79.196z" fill="currentColor"></path></svg></span></a>
	<span>Choosing a metric for your task
	</span></h1>

<p><strong>So you’ve trained your model and want to see how well it’s doing on a dataset of your choice. Where do you start?</strong></p>
<p>There is no “one size fits all” approach to choosing an evaluation metric, but some good guidelines to keep in mind are:</p>
<h2 class="relative group"><a id="categories-of-metrics" class="header-link block pr-1.5 text-lg no-hover:hidden with-hover:absolute with-hover:p-1.5 with-hover:opacity-0 with-hover:group-hover:opacity-100 with-hover:right-full" href="#categories-of-metrics"><span><svg class="" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" aria-hidden="true" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 256 256"><path d="M167.594 88.393a8.001 8.001 0 0 1 0 11.314l-67.882 67.882a8 8 0 1 1-11.314-11.315l67.882-67.881a8.003 8.003 0 0 1 11.314 0zm-28.287 84.86l-28.284 28.284a40 40 0 0 1-56.567-56.567l28.284-28.284a8 8 0 0 0-11.315-11.315l-28.284 28.284a56 56 0 0 0 79.196 79.197l28.285-28.285a8 8 0 1 0-11.315-11.314zM212.852 43.14a56.002 56.002 0 0 0-79.196 0l-28.284 28.284a8 8 0 1 0 11.314 11.314l28.284-28.284a40 40 0 0 1 56.568 56.567l-28.285 28.285a8 8 0 0 0 11.315 11.314l28.284-28.284a56.065 56.065 0 0 0 0-79.196z" fill="currentColor"></path></svg></span></a>
	<span>Categories of metrics
	</span></h2>

<p>There are 3 high-level categories of metrics:</p>
<ol><li><em>Generic metrics</em>, which can be applied to a variety of situations and datasets, such as precision and accuracy.</li>
<li><em>Task-specific metrics</em>, which are limited to a given task, such as Machine Translation (often evaluated using metrics <a href="https://huggingface.co/metrics/bleu" rel="nofollow">BLEU</a> or <a href="https://huggingface.co/metrics/rouge" rel="nofollow">ROUGE</a>) or Named Entity Recognition (often evaluated with <a href="https://huggingface.co/metrics/seqeval" rel="nofollow">seqeval</a>).</li>
<li><em>Dataset-specific metrics</em>, which aim to measure model performance on specific benchmarks: for instance, the <a href="https://huggingface.co/datasets/glue" rel="nofollow">GLUE benchmark</a> has a dedicated <a href="https://huggingface.co/metrics/glue" rel="nofollow">evaluation metric</a>.</li></ol>
<p>Let’s look at each of these three cases:</p>
<h3 class="relative group"><a id="generic-metrics" class="header-link block pr-1.5 text-lg no-hover:hidden with-hover:absolute with-hover:p-1.5 with-hover:opacity-0 with-hover:group-hover:opacity-100 with-hover:right-full" href="#generic-metrics"><span><svg class="" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" aria-hidden="true" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 256 256"><path d="M167.594 88.393a8.001 8.001 0 0 1 0 11.314l-67.882 67.882a8 8 0 1 1-11.314-11.315l67.882-67.881a8.003 8.003 0 0 1 11.314 0zm-28.287 84.86l-28.284 28.284a40 40 0 0 1-56.567-56.567l28.284-28.284a8 8 0 0 0-11.315-11.315l-28.284 28.284a56 56 0 0 0 79.196 79.197l28.285-28.285a8 8 0 1 0-11.315-11.314zM212.852 43.14a56.002 56.002 0 0 0-79.196 0l-28.284 28.284a8 8 0 1 0 11.314 11.314l28.284-28.284a40 40 0 0 1 56.568 56.567l-28.285 28.285a8 8 0 0 0 11.315 11.314l28.284-28.284a56.065 56.065 0 0 0 0-79.196z" fill="currentColor"></path></svg></span></a>
	<span>Generic metrics
	</span></h3>

<p>Many of the metrics used in the Machine Learning community are quite generic and can be applied in a variety of tasks and datasets.</p>
<p>This is the case for metrics like <a href="https://huggingface.co/metrics/accuracy" rel="nofollow">accuracy</a> and <a href="https://huggingface.co/metrics/precision" rel="nofollow">precision</a>, which can be used for evaluating labeled (supervised) datasets, as well as <a href="https://huggingface.co/metrics/perplexity" rel="nofollow">perplexity</a>, which can be used for evaluating different kinds of (unsupervised) generative tasks.</p>
<p>To see the input structure of a given metric, you can look at its metric card. For example, in the case of <a href="https://huggingface.co/metrics/precision" rel="nofollow">precision</a>, the format is:</p>

	<div class="code-block relative"><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5   text-gray-600 " title="code excerpt" type="button"><svg class="" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg>
	<div class="absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0"><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent; "></div>
	Copied</div></button></div>
	<pre><!-- HTML_TAG_START --><span class="hljs-meta">&gt;&gt;&gt;</span> <span class="language-python">precision_metric = evaluate.load(<span class="hljs-string">&quot;precision&quot;</span>)</span>
<span class="hljs-meta">&gt;&gt;&gt;</span> <span class="language-python">results = precision_metric.compute(references=[<span class="hljs-number">0</span>, <span class="hljs-number">1</span>], predictions=[<span class="hljs-number">0</span>, <span class="hljs-number">1</span>])</span>
<span class="hljs-meta">&gt;&gt;&gt;</span> <span class="language-python"><span class="hljs-built_in">print</span>(results)</span>
{&#x27;precision&#x27;: 1.0}<!-- HTML_TAG_END --></pre></div>
<h3 class="relative group"><a id="taskspecific-metrics" class="header-link block pr-1.5 text-lg no-hover:hidden with-hover:absolute with-hover:p-1.5 with-hover:opacity-0 with-hover:group-hover:opacity-100 with-hover:right-full" href="#taskspecific-metrics"><span><svg class="" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" aria-hidden="true" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 256 256"><path d="M167.594 88.393a8.001 8.001 0 0 1 0 11.314l-67.882 67.882a8 8 0 1 1-11.314-11.315l67.882-67.881a8.003 8.003 0 0 1 11.314 0zm-28.287 84.86l-28.284 28.284a40 40 0 0 1-56.567-56.567l28.284-28.284a8 8 0 0 0-11.315-11.315l-28.284 28.284a56 56 0 0 0 79.196 79.197l28.285-28.285a8 8 0 1 0-11.315-11.314zM212.852 43.14a56.002 56.002 0 0 0-79.196 0l-28.284 28.284a8 8 0 1 0 11.314 11.314l28.284-28.284a40 40 0 0 1 56.568 56.567l-28.285 28.285a8 8 0 0 0 11.315 11.314l28.284-28.284a56.065 56.065 0 0 0 0-79.196z" fill="currentColor"></path></svg></span></a>
	<span>Task-specific metrics
	</span></h3>

<p>Popular ML tasks like Machine Translation and Named Entity Recognition have specific metrics that can be used to compare models. For example, a series of different metrics have been proposed for text generation, ranging from <a href="https://huggingface.co/metrics/bleu" rel="nofollow">BLEU</a> and its derivatives such as <a href="https://huggingface.co/metrics/google_bleu" rel="nofollow">GoogleBLEU</a> and <a href="https://huggingface.co/metrics/gleu" rel="nofollow">GLEU</a>, but also <a href="https://huggingface.co/metrics/rouge" rel="nofollow">ROUGE</a>, <a href="https://huggingface.co/metrics/mauve" rel="nofollow">MAUVE</a>, etc.</p>
<p>You can find the right metric for your task by:</p>
<ul><li><strong>Looking at the <a href="https://huggingface.co/tasks" rel="nofollow">Task pages</a></strong> to see what metrics can be used for evaluating models for a given task.</li>
<li><strong>Checking out leaderboards</strong> on sites like <a href="https://paperswithcode.com/" rel="nofollow">Papers With Code</a> (you can search by task and by dataset).</li>
<li><strong>Reading the metric cards</strong> for the relevant metrics and see which ones are a good fit for your use case. For example, see the <a href="https://github.com/huggingface/evaluate/tree/main/metrics/bleu" rel="nofollow">BLEU metric card</a> or <a href="https://github.com/huggingface/evaluate/tree/main/metrics/squad" rel="nofollow">SQuaD metric card</a>.</li>
<li><strong>Looking at papers and blog posts</strong> published on the topic and see what metrics they report. This can change over time, so try to pick papers from the last couple of years!</li></ul>
<h3 class="relative group"><a id="datasetspecific-metrics" class="header-link block pr-1.5 text-lg no-hover:hidden with-hover:absolute with-hover:p-1.5 with-hover:opacity-0 with-hover:group-hover:opacity-100 with-hover:right-full" href="#datasetspecific-metrics"><span><svg class="" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" aria-hidden="true" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 256 256"><path d="M167.594 88.393a8.001 8.001 0 0 1 0 11.314l-67.882 67.882a8 8 0 1 1-11.314-11.315l67.882-67.881a8.003 8.003 0 0 1 11.314 0zm-28.287 84.86l-28.284 28.284a40 40 0 0 1-56.567-56.567l28.284-28.284a8 8 0 0 0-11.315-11.315l-28.284 28.284a56 56 0 0 0 79.196 79.197l28.285-28.285a8 8 0 1 0-11.315-11.314zM212.852 43.14a56.002 56.002 0 0 0-79.196 0l-28.284 28.284a8 8 0 1 0 11.314 11.314l28.284-28.284a40 40 0 0 1 56.568 56.567l-28.285 28.285a8 8 0 0 0 11.315 11.314l28.284-28.284a56.065 56.065 0 0 0 0-79.196z" fill="currentColor"></path></svg></span></a>
	<span>Dataset-specific metrics
	</span></h3>

<p>Some datasets have specific metrics associated with them — this is especially in the case of popular benchmarks like <a href="https://huggingface.co/metrics/glue" rel="nofollow">GLUE</a> and <a href="https://huggingface.co/metrics/squad" rel="nofollow">SQuAD</a>.</p>


<div class="course-tip course-tip-orange bg-gradient-to-br dark:bg-gradient-to-r before:border-orange-500 dark:before:border-orange-800 from-orange-50 dark:from-gray-900 to-white dark:to-gray-950 border border-orange-50 text-orange-700 dark:text-gray-400">💡
GLUE is actually a collection of different subsets on different tasks, so first you need to choose the one that corresponds to the NLI task, such as mnli, which is described as “crowdsourced collection of sentence pairs with textual entailment annotations”
</div>
<p>If you are evaluating your model on a benchmark dataset like the ones mentioned above, you can use its dedicated evaluation metric. Make sure you respect the format that they require. For example, to evaluate your model on the <a href="https://huggingface.co/datasets/squad" rel="nofollow">SQuAD</a> dataset, you need to feed the <code>question</code> and <code>context</code> into your model and return the <code>prediction_text</code>, which should be compared with the <code>references</code> (based on matching the <code>id</code> of the question) :</p>

	<div class="code-block relative"><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5   text-gray-600 " title="code excerpt" type="button"><svg class="" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg>
	<div class="absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0"><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent; "></div>
	Copied</div></button></div>
	<pre><!-- HTML_TAG_START --><span class="hljs-meta">&gt;&gt;&gt;</span> <span class="language-python"><span class="hljs-keyword">from</span> evaluate <span class="hljs-keyword">import</span> load</span>
<span class="hljs-meta">&gt;&gt;&gt;</span> <span class="language-python">squad_metric = load(<span class="hljs-string">&quot;squad&quot;</span>)</span>
<span class="hljs-meta">&gt;&gt;&gt;</span> <span class="language-python">predictions = [{<span class="hljs-string">&#x27;prediction_text&#x27;</span>: <span class="hljs-string">&#x27;1976&#x27;</span>, <span class="hljs-string">&#x27;id&#x27;</span>: <span class="hljs-string">&#x27;56e10a3be3433e1400422b22&#x27;</span>}]</span>
<span class="hljs-meta">&gt;&gt;&gt;</span> <span class="language-python">references = [{<span class="hljs-string">&#x27;answers&#x27;</span>: {<span class="hljs-string">&#x27;answer_start&#x27;</span>: [<span class="hljs-number">97</span>], <span class="hljs-string">&#x27;text&#x27;</span>: [<span class="hljs-string">&#x27;1976&#x27;</span>]}, <span class="hljs-string">&#x27;id&#x27;</span>: <span class="hljs-string">&#x27;56e10a3be3433e1400422b22&#x27;</span>}]</span>
<span class="hljs-meta">&gt;&gt;&gt;</span> <span class="language-python">results = squad_metric.compute(predictions=predictions, references=references)</span>
<span class="hljs-meta">&gt;&gt;&gt;</span> <span class="language-python">results</span>
{&#x27;exact_match&#x27;: 100.0, &#x27;f1&#x27;: 100.0}<!-- HTML_TAG_END --></pre></div>
<p>You can find examples of dataset structures by consulting the “Dataset Preview” function or the dataset card for a given dataset, and you can see how to use its dedicated evaluation function based on the metric card.</p>


		<script type="module" data-hydrate="u6ql3f">
		import { start } from "/docs/evaluate/pr_36/en/_app/start-hf-doc-builder.js";
		start({
			target: document.querySelector('[data-hydrate="u6ql3f"]').parentNode,
			paths: {"base":"/docs/evaluate/pr_36/en","assets":"/docs/evaluate/pr_36/en"},
			session: {},
			route: false,
			spa: false,
			trailing_slash: "never",
			hydrate: {
				status: 200,
				error: null,
				nodes: [
					import("/docs/evaluate/pr_36/en/_app/pages/__layout.svelte-hf-doc-builder.js"),
						import("/docs/evaluate/pr_36/en/_app/pages/choosing_a_metric.mdx-hf-doc-builder.js")
				],
				params: {}
			}
		});
	</script>
