<meta charset="utf-8" /><meta http-equiv="content-security-policy" content=""><meta name="hf:doc:metadata" content="{&quot;local&quot;:&quot;types-of-evaluations-in-evaluate&quot;,&quot;sections&quot;:[{&quot;local&quot;:&quot;metrics&quot;,&quot;title&quot;:&quot;Metrics&quot;},{&quot;local&quot;:&quot;comparisons&quot;,&quot;title&quot;:&quot;Comparisons&quot;},{&quot;local&quot;:&quot;measurements&quot;,&quot;title&quot;:&quot;Measurements&quot;}],&quot;title&quot;:&quot;Types of Evaluations in 🤗 Evaluate&quot;}" data-svelte="svelte-1phssyn">
	<link rel="modulepreload" href="/docs/evaluate/pr_299/en/_app/assets/pages/__layout.svelte-hf-doc-builder.css">
	<link rel="modulepreload" href="/docs/evaluate/pr_299/en/_app/start-hf-doc-builder.js">
	<link rel="modulepreload" href="/docs/evaluate/pr_299/en/_app/chunks/vendor-hf-doc-builder.js">
	<link rel="modulepreload" href="/docs/evaluate/pr_299/en/_app/chunks/paths-hf-doc-builder.js">
	<link rel="modulepreload" href="/docs/evaluate/pr_299/en/_app/pages/__layout.svelte-hf-doc-builder.js">
	<link rel="modulepreload" href="/docs/evaluate/pr_299/en/_app/pages/types_of_evaluations.mdx-hf-doc-builder.js">
	<link rel="modulepreload" href="/docs/evaluate/pr_299/en/_app/chunks/IconCopyLink-hf-doc-builder.js"> 





<h1 class="relative group"><a id="types-of-evaluations-in-evaluate" class="header-link block pr-1.5 text-lg no-hover:hidden with-hover:absolute with-hover:p-1.5 with-hover:opacity-0 with-hover:group-hover:opacity-100 with-hover:right-full" href="#types-of-evaluations-in-evaluate"><span><svg class="" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" aria-hidden="true" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 256 256"><path d="M167.594 88.393a8.001 8.001 0 0 1 0 11.314l-67.882 67.882a8 8 0 1 1-11.314-11.315l67.882-67.881a8.003 8.003 0 0 1 11.314 0zm-28.287 84.86l-28.284 28.284a40 40 0 0 1-56.567-56.567l28.284-28.284a8 8 0 0 0-11.315-11.315l-28.284 28.284a56 56 0 0 0 79.196 79.197l28.285-28.285a8 8 0 1 0-11.315-11.314zM212.852 43.14a56.002 56.002 0 0 0-79.196 0l-28.284 28.284a8 8 0 1 0 11.314 11.314l28.284-28.284a40 40 0 0 1 56.568 56.567l-28.285 28.285a8 8 0 0 0 11.315 11.314l28.284-28.284a56.065 56.065 0 0 0 0-79.196z" fill="currentColor"></path></svg></span></a>
	<span>Types of Evaluations in 🤗 Evaluate
	</span></h1>

<p>The goal of the 🤗 Evaluate library is to support different types of evaluation, depending on different goals, datasets and models.</p>
<p>Here are the types of evaluations that are currently supported with a few examples for each:</p>
<h2 class="relative group"><a id="metrics" class="header-link block pr-1.5 text-lg no-hover:hidden with-hover:absolute with-hover:p-1.5 with-hover:opacity-0 with-hover:group-hover:opacity-100 with-hover:right-full" href="#metrics"><span><svg class="" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" aria-hidden="true" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 256 256"><path d="M167.594 88.393a8.001 8.001 0 0 1 0 11.314l-67.882 67.882a8 8 0 1 1-11.314-11.315l67.882-67.881a8.003 8.003 0 0 1 11.314 0zm-28.287 84.86l-28.284 28.284a40 40 0 0 1-56.567-56.567l28.284-28.284a8 8 0 0 0-11.315-11.315l-28.284 28.284a56 56 0 0 0 79.196 79.197l28.285-28.285a8 8 0 1 0-11.315-11.314zM212.852 43.14a56.002 56.002 0 0 0-79.196 0l-28.284 28.284a8 8 0 1 0 11.314 11.314l28.284-28.284a40 40 0 0 1 56.568 56.567l-28.285 28.285a8 8 0 0 0 11.315 11.314l28.284-28.284a56.065 56.065 0 0 0 0-79.196z" fill="currentColor"></path></svg></span></a>
	<span>Metrics
	</span></h2>

A metric measures the performance of a model on a given dataset. This is often based on an existing ground truth (i.e. a set of references), but there are also *referenceless metrics* which allow evaluating generated text by leveraging a pretrained model such as [GPT-2](https://huggingface.co/gpt2).
<p>Examples of metrics include:</p>
<ul><li><a href="https://huggingface.co/metrics/accuracy" rel="nofollow">Accuracy</a> : the proportion of correct predictions among the total number of cases processed.</li>
<li><a href="https://huggingface.co/metrics/exact_match" rel="nofollow">Exact Match</a>: the rate at which the input predicted strings exactly match their references.</li>
<li><a href="https://huggingface.co/metrics/mean_iou" rel="nofollow">Mean Intersection over union (IoUO)</a>: the area of overlap between the predicted segmentation of an image and the ground truth divided by the area of union between the predicted segmentation and the ground truth.</li></ul>
<p>Metrics are often used to track model performance on benchmark datasets, and to report progress on tasks such as <a href="https://huggingface.co/tasks/translation" rel="nofollow">machine translation</a> and <a href="https://huggingface.co/tasks/image-classification" rel="nofollow">image classification</a>.</p>
<h2 class="relative group"><a id="comparisons" class="header-link block pr-1.5 text-lg no-hover:hidden with-hover:absolute with-hover:p-1.5 with-hover:opacity-0 with-hover:group-hover:opacity-100 with-hover:right-full" href="#comparisons"><span><svg class="" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" aria-hidden="true" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 256 256"><path d="M167.594 88.393a8.001 8.001 0 0 1 0 11.314l-67.882 67.882a8 8 0 1 1-11.314-11.315l67.882-67.881a8.003 8.003 0 0 1 11.314 0zm-28.287 84.86l-28.284 28.284a40 40 0 0 1-56.567-56.567l28.284-28.284a8 8 0 0 0-11.315-11.315l-28.284 28.284a56 56 0 0 0 79.196 79.197l28.285-28.285a8 8 0 1 0-11.315-11.314zM212.852 43.14a56.002 56.002 0 0 0-79.196 0l-28.284 28.284a8 8 0 1 0 11.314 11.314l28.284-28.284a40 40 0 0 1 56.568 56.567l-28.285 28.285a8 8 0 0 0 11.315 11.314l28.284-28.284a56.065 56.065 0 0 0 0-79.196z" fill="currentColor"></path></svg></span></a>
	<span>Comparisons
	</span></h2>

<p>Comparisons can be useful to compare the performance of two or more models on a single test dataset.</p>
<p>For instance, the <a href="https://github.com/huggingface/evaluate/tree/main/comparisons/mcnemar" rel="nofollow">McNemar Test</a> is a paired nonparametric statistical hypothesis test that takes the predictions of two models and compares them, aiming to measure whether the models’s predictions diverge or not. The p value it outputs, which ranges from <code>0.0</code> to <code>1.0</code>, indicates the difference between the two models’ predictions, with a lower p value indicating a more significant difference.</p>
<p>Comparisons have yet to be systematically used when comparing and reporting model performance, however they are useful tools to go beyond simply comparing leaderboard scores and for getting more information on the way model prediction differ.</p>
<h2 class="relative group"><a id="measurements" class="header-link block pr-1.5 text-lg no-hover:hidden with-hover:absolute with-hover:p-1.5 with-hover:opacity-0 with-hover:group-hover:opacity-100 with-hover:right-full" href="#measurements"><span><svg class="" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" aria-hidden="true" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 256 256"><path d="M167.594 88.393a8.001 8.001 0 0 1 0 11.314l-67.882 67.882a8 8 0 1 1-11.314-11.315l67.882-67.881a8.003 8.003 0 0 1 11.314 0zm-28.287 84.86l-28.284 28.284a40 40 0 0 1-56.567-56.567l28.284-28.284a8 8 0 0 0-11.315-11.315l-28.284 28.284a56 56 0 0 0 79.196 79.197l28.285-28.285a8 8 0 1 0-11.315-11.314zM212.852 43.14a56.002 56.002 0 0 0-79.196 0l-28.284 28.284a8 8 0 1 0 11.314 11.314l28.284-28.284a40 40 0 0 1 56.568 56.567l-28.285 28.285a8 8 0 0 0 11.315 11.314l28.284-28.284a56.065 56.065 0 0 0 0-79.196z" fill="currentColor"></path></svg></span></a>
	<span>Measurements
	</span></h2>

<p>In the 🤗 Evaluate library, measurements are tools for gaining more insights on datasets and model predictions.</p>
<p>For instance, in the case of datasets, it can be useful to calculate the <a href="https://github.com/huggingface/evaluate/tree/main/measurements/word_length" rel="nofollow">average word length</a> of a dataset’s entries, and how it is distributed — this can help when choosing the maximum input length for <a href="https://huggingface.co/docs/transformers/main_classes/tokenizer" rel="nofollow">Tokenizer</a>.</p>
<p>In the case of model predictions, it can help to calculate the average <a href="https://huggingface.co/metrics/perplexity" rel="nofollow">perplexity</a> of model predictions using different models such as <a href="https://huggingface.co/gpt2" rel="nofollow">GPT-2</a> and <a href="https://huggingface.co/bert-base-uncased" rel="nofollow">BERT</a>, which can indicate the quality of generated text when no reference is available.</p>
<p>All three types of evaluation supported by the 🤗 Evaluate library are meant to be mutually complementary, and help our community carry out more mindful and responsible evaluation.</p>
<p>We will continue adding more types of metrics, measurements and comparisons in coming months, and are counting on community involvement (via <a href="https://github.com/huggingface/evaluate/compare" rel="nofollow">PRs</a> and <a href="https://github.com/huggingface/evaluate/issues/new/choose" rel="nofollow">issues</a>) to make the library as extensive and inclusive as possible!</p>


		<script type="module" data-hydrate="17lfe6c">
		import { start } from "/docs/evaluate/pr_299/en/_app/start-hf-doc-builder.js";
		start({
			target: document.querySelector('[data-hydrate="17lfe6c"]').parentNode,
			paths: {"base":"/docs/evaluate/pr_299/en","assets":"/docs/evaluate/pr_299/en"},
			session: {},
			route: false,
			spa: false,
			trailing_slash: "never",
			hydrate: {
				status: 200,
				error: null,
				nodes: [
					import("/docs/evaluate/pr_299/en/_app/pages/__layout.svelte-hf-doc-builder.js"),
						import("/docs/evaluate/pr_299/en/_app/pages/types_of_evaluations.mdx-hf-doc-builder.js")
				],
				params: {}
			}
		});
	</script>
