<meta charset="utf-8" /><meta http-equiv="content-security-policy" content=""><meta name="hf:doc:metadata" content="{&quot;local&quot;:&quot;all-about-metrics&quot;,&quot;sections&quot;:[{&quot;local&quot;:&quot;eli5-load&quot;,&quot;title&quot;:&quot;ELI5: `load`&quot;},{&quot;local&quot;:&quot;distributed-evaluation&quot;,&quot;title&quot;:&quot;Distributed evaluation&quot;}],&quot;title&quot;:&quot;All about metrics&quot;}" data-svelte="svelte-1phssyn">
	<link rel="modulepreload" href="/docs/evaluate/pr_56/en/_app/assets/pages/__layout.svelte-7fcec71d.css">
	<link rel="modulepreload" href="/docs/evaluate/pr_56/en/_app/start-93356680.js">
	<link rel="modulepreload" href="/docs/evaluate/pr_56/en/_app/chunks/vendor-e830fc9c.js">
	<link rel="modulepreload" href="/docs/evaluate/pr_56/en/_app/chunks/paths-4b3c6e7e.js">
	<link rel="modulepreload" href="/docs/evaluate/pr_56/en/_app/pages/__layout.svelte-2159ccac.js">
	<link rel="modulepreload" href="/docs/evaluate/pr_56/en/_app/pages/about_metrics.mdx-efa40886.js">
	<link rel="modulepreload" href="/docs/evaluate/pr_56/en/_app/chunks/IconCopyLink-af4c5c5b.js"> 





<h1 class="relative group"><a id="all-about-metrics" class="header-link block pr-1.5 text-lg no-hover:hidden with-hover:absolute with-hover:p-1.5 with-hover:opacity-0 with-hover:group-hover:opacity-100 with-hover:right-full" href="#all-about-metrics"><span><svg class="" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" aria-hidden="true" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 256 256"><path d="M167.594 88.393a8.001 8.001 0 0 1 0 11.314l-67.882 67.882a8 8 0 1 1-11.314-11.315l67.882-67.881a8.003 8.003 0 0 1 11.314 0zm-28.287 84.86l-28.284 28.284a40 40 0 0 1-56.567-56.567l28.284-28.284a8 8 0 0 0-11.315-11.315l-28.284 28.284a56 56 0 0 0 79.196 79.197l28.285-28.285a8 8 0 1 0-11.315-11.314zM212.852 43.14a56.002 56.002 0 0 0-79.196 0l-28.284 28.284a8 8 0 1 0 11.314 11.314l28.284-28.284a40 40 0 0 1 56.568 56.567l-28.285 28.285a8 8 0 0 0 11.315 11.314l28.284-28.284a56.065 56.065 0 0 0 0-79.196z" fill="currentColor"></path></svg></span></a>
	<span>All about metrics
	</span></h1>

<p>🤗 Datasets provides access to a wide range of NLP metrics. You can load metrics associated with benchmark datasets like GLUE or SQuAD, and complex metrics like BLEURT or BERTScore, with a single command: <a href="/docs/evaluate/pr_56/en/package_reference/loading_methods#evaluate.load">load()</a>. Once you’ve loaded a metric, easily compute and evaluate a model’s performance.</p>
<h2 class="relative group"><a id="eli5-load" class="header-link block pr-1.5 text-lg no-hover:hidden with-hover:absolute with-hover:p-1.5 with-hover:opacity-0 with-hover:group-hover:opacity-100 with-hover:right-full" href="#eli5-load"><span><svg class="" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" aria-hidden="true" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 256 256"><path d="M167.594 88.393a8.001 8.001 0 0 1 0 11.314l-67.882 67.882a8 8 0 1 1-11.314-11.315l67.882-67.881a8.003 8.003 0 0 1 11.314 0zm-28.287 84.86l-28.284 28.284a40 40 0 0 1-56.567-56.567l28.284-28.284a8 8 0 0 0-11.315-11.315l-28.284 28.284a56 56 0 0 0 79.196 79.197l28.285-28.285a8 8 0 1 0-11.315-11.314zM212.852 43.14a56.002 56.002 0 0 0-79.196 0l-28.284 28.284a8 8 0 1 0 11.314 11.314l28.284-28.284a40 40 0 0 1 56.568 56.567l-28.285 28.285a8 8 0 0 0 11.315 11.314l28.284-28.284a56.065 56.065 0 0 0 0-79.196z" fill="currentColor"></path></svg></span></a>
	<span>ELI5: <code>load</code></span></h2>

<p>Loading a dataset and loading a metric share many similarities. This was an intentional design choice because we wanted to create a simple and unified experience. When you call <a href="/docs/evaluate/pr_56/en/package_reference/loading_methods#evaluate.load">load()</a>, the metric loading script is downloaded and imported from GitHub (if it hasn’t already been downloaded before). It contains information about the metric such as it’s citation, homepage, and description.</p>
<p>The metric loading script will instantiate and return a <code>Metric</code> object. This stores the predictions and references, which you need to compute the metric values. The <code>Metric</code> object is stored as an Apache Arrow table. As a result, the predictions and references are stored directly on disk with memory-mapping. This enables 🤗 Datasets to do a lazy computation of the metric, and makes it easier to gather all the predictions in a distributed setting.</p>
<h2 class="relative group"><a id="distributed-evaluation" class="header-link block pr-1.5 text-lg no-hover:hidden with-hover:absolute with-hover:p-1.5 with-hover:opacity-0 with-hover:group-hover:opacity-100 with-hover:right-full" href="#distributed-evaluation"><span><svg class="" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" aria-hidden="true" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 256 256"><path d="M167.594 88.393a8.001 8.001 0 0 1 0 11.314l-67.882 67.882a8 8 0 1 1-11.314-11.315l67.882-67.881a8.003 8.003 0 0 1 11.314 0zm-28.287 84.86l-28.284 28.284a40 40 0 0 1-56.567-56.567l28.284-28.284a8 8 0 0 0-11.315-11.315l-28.284 28.284a56 56 0 0 0 79.196 79.197l28.285-28.285a8 8 0 1 0-11.315-11.314zM212.852 43.14a56.002 56.002 0 0 0-79.196 0l-28.284 28.284a8 8 0 1 0 11.314 11.314l28.284-28.284a40 40 0 0 1 56.568 56.567l-28.285 28.285a8 8 0 0 0 11.315 11.314l28.284-28.284a56.065 56.065 0 0 0 0-79.196z" fill="currentColor"></path></svg></span></a>
	<span>Distributed evaluation
	</span></h2>

<p>Computing metrics in a distributed environment can be tricky. Metric evaluation is executed in separate Python processes, or nodes, on different subsets of a dataset. Typically, when a metric score is additive (<code>f(AuB) = f(A) + f(B)</code>), you can use distributed reduce operations to gather the scores for each subset of the dataset. But when a metric is non-additive (<code>f(AuB) ≠ f(A) + f(B)</code>), it’s not that simple. For example, you can’t take the sum of the <a href="https://huggingface.co/metrics/f1" rel="nofollow">F1</a> scores of each data subset as your <strong>final metric</strong>.</p>
<p>A common way to overcome this issue is to fallback on single process evaluation. The metrics are evaluated on a single GPU, which becomes inefficient.</p>
<p>🤗 Datasets solves this issue by only computing the final metric on the first node. The predictions and references are computed and provided to the metric separately for each node. These are temporarily stored in an Apache Arrow table, avoiding cluttering the GPU or CPU memory. When you are ready to <code>Metric.compute</code> the final metric, the first node is able to access the predictions and references stored on all the other nodes. Once it has gathered all the predictions and references, <code>Metric.compute</code> will perform the final metric evaluation.</p>
<p>This solution allows 🤗 Datasets to perform distributed predictions, which is important for evaluation speed in distributed settings. At the same time, you can also use complex non-additive metrics without wasting valuable GPU or CPU memory.</p>


		<script type="module" data-hydrate="1nwaq7g">
		import { start } from "/docs/evaluate/pr_56/en/_app/start-93356680.js";
		start({
			target: document.querySelector('[data-hydrate="1nwaq7g"]').parentNode,
			paths: {"base":"/docs/evaluate/pr_56/en","assets":"/docs/evaluate/pr_56/en"},
			session: {},
			route: false,
			spa: false,
			trailing_slash: "never",
			hydrate: {
				status: 200,
				error: null,
				nodes: [
					import("/docs/evaluate/pr_56/en/_app/pages/__layout.svelte-2159ccac.js"),
						import("/docs/evaluate/pr_56/en/_app/pages/about_metrics.mdx-efa40886.js")
				],
				params: {}
			}
		});
	</script>
