<meta charset="utf-8" /><meta http-equiv="content-security-policy" content=""><meta name="hf:doc:metadata" content="{&quot;local&quot;:&quot;optimum-inference-with-openvino&quot;,&quot;sections&quot;:[{&quot;local&quot;:&quot;switching-from-transformers-to-optimum-inference&quot;,&quot;title&quot;:&quot;Switching from Transformers to Optimum Inference&quot;},{&quot;local&quot;:&quot;run-inference-for-sequencetosequence-models&quot;,&quot;title&quot;:&quot;Run inference for sequence-to-sequence models&quot;}],&quot;title&quot;:&quot;Optimum Inference with OpenVINO&quot;}" data-svelte="svelte-1phssyn">
	<link rel="modulepreload" href="/docs/optimum.intel/pr_439/en/_app/assets/pages/__layout.svelte-hf-doc-builder.css">
	<link rel="modulepreload" href="/docs/optimum.intel/pr_439/en/_app/start-hf-doc-builder.js">
	<link rel="modulepreload" href="/docs/optimum.intel/pr_439/en/_app/chunks/vendor-hf-doc-builder.js">
	<link rel="modulepreload" href="/docs/optimum.intel/pr_439/en/_app/chunks/paths-hf-doc-builder.js">
	<link rel="modulepreload" href="/docs/optimum.intel/pr_439/en/_app/pages/__layout.svelte-hf-doc-builder.js">
	<link rel="modulepreload" href="/docs/optimum.intel/pr_439/en/_app/pages/inference.mdx-hf-doc-builder.js">
	<link rel="modulepreload" href="/docs/optimum.intel/pr_439/en/_app/chunks/IconCopyLink-hf-doc-builder.js">
	<link rel="modulepreload" href="/docs/optimum.intel/pr_439/en/_app/chunks/CodeBlock-hf-doc-builder.js"> 






<h1 class="relative group"><a id="optimum-inference-with-openvino" class="header-link block pr-1.5 text-lg no-hover:hidden with-hover:absolute with-hover:p-1.5 with-hover:opacity-0 with-hover:group-hover:opacity-100 with-hover:right-full" href="#optimum-inference-with-openvino"><span><svg class="" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" aria-hidden="true" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 256 256"><path d="M167.594 88.393a8.001 8.001 0 0 1 0 11.314l-67.882 67.882a8 8 0 1 1-11.314-11.315l67.882-67.881a8.003 8.003 0 0 1 11.314 0zm-28.287 84.86l-28.284 28.284a40 40 0 0 1-56.567-56.567l28.284-28.284a8 8 0 0 0-11.315-11.315l-28.284 28.284a56 56 0 0 0 79.196 79.197l28.285-28.285a8 8 0 1 0-11.315-11.314zM212.852 43.14a56.002 56.002 0 0 0-79.196 0l-28.284 28.284a8 8 0 1 0 11.314 11.314l28.284-28.284a40 40 0 0 1 56.568 56.567l-28.285 28.285a8 8 0 0 0 11.315 11.314l28.284-28.284a56.065 56.065 0 0 0 0-79.196z" fill="currentColor"></path></svg></span></a>
	<span>Optimum Inference with OpenVINO
	</span></h1>

<p>Optimum Intel can be used to load optimized models from the <a href="hf.co/models">Hugging Face Hub</a> and create pipelines to run inference with OpenVINO Runtime without rewriting your APIs.</p>
<h2 class="relative group"><a id="switching-from-transformers-to-optimum-inference" class="header-link block pr-1.5 text-lg no-hover:hidden with-hover:absolute with-hover:p-1.5 with-hover:opacity-0 with-hover:group-hover:opacity-100 with-hover:right-full" href="#switching-from-transformers-to-optimum-inference"><span><svg class="" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" aria-hidden="true" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 256 256"><path d="M167.594 88.393a8.001 8.001 0 0 1 0 11.314l-67.882 67.882a8 8 0 1 1-11.314-11.315l67.882-67.881a8.003 8.003 0 0 1 11.314 0zm-28.287 84.86l-28.284 28.284a40 40 0 0 1-56.567-56.567l28.284-28.284a8 8 0 0 0-11.315-11.315l-28.284 28.284a56 56 0 0 0 79.196 79.197l28.285-28.285a8 8 0 1 0-11.315-11.314zM212.852 43.14a56.002 56.002 0 0 0-79.196 0l-28.284 28.284a8 8 0 1 0 11.314 11.314l28.284-28.284a40 40 0 0 1 56.568 56.567l-28.285 28.285a8 8 0 0 0 11.315 11.314l28.284-28.284a56.065 56.065 0 0 0 0-79.196z" fill="currentColor"></path></svg></span></a>
	<span>Switching from Transformers to Optimum Inference
	</span></h2>

<p>The Optimum Inference models are API compatible with Hugging Face Transformers models. This means you can just replace your <code>AutoModelForXxx</code> class with the corresponding <code>OVModelForXxx</code> class.
To perform inference from a vanilla Transformers models, which will be later converted to an OpenVINO IR, set <code>from_transformers=True</code> when loading the model with the <code>from_pretrained()</code> method. </p>
<p>Here is an example on how to perform inference with OpenVINO Runtime for a text classification class:</p>

	<div class="code-block relative"><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5   text-gray-600 " title="code excerpt" type="button"><svg class="" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg>
	<div class="absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0"><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent; "></div>
	Copied</div></button></div>
	<pre><!-- HTML_TAG_START --><span class="hljs-deletion">-from transformers import AutoModelForSequenceClassification</span>
<span class="hljs-addition">+from optimum.intel.openvino import OVModelForSequenceClassification</span>
from transformers import AutoTokenizer, pipeline
model_id = &quot;distilbert-base-uncased-finetuned-sst-2-english&quot;
<span class="hljs-deletion">-model = AutoModelForSequenceClassification.from_pretrained(model_id)</span>
<span class="hljs-addition">+model = OVModelForSequenceClassification.from_pretrained(model_id, from_transformers=True)</span>
tokenizer = AutoTokenizer.from_pretrained(model_id)
cls_pipe = pipeline(&quot;text-classification&quot;, model=model, tokenizer=tokenizer)
result = cls_pipe(&quot;He&#x27;s a dreadful magician.&quot;)

[{&#x27;label&#x27;: &#x27;NEGATIVE&#x27;, &#x27;score&#x27;: 0.9919503927230835}]<!-- HTML_TAG_END --></pre></div>
<p>An important thing to mention is that the model is compiled just before the first inference, which will inflate the latency of the first inference.</p>
<p>To easily save the resulting model, you can use the <code>save_pretrained()</code> method, which will save both the BIN and XML files describing the graph.</p>

	<div class="code-block relative"><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5   text-gray-600 " title="code excerpt" type="button"><svg class="" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg>
	<div class="absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0"><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent; "></div>
	Copied</div></button></div>
	<pre><!-- HTML_TAG_START --><span class="hljs-comment"># Save the exported model</span>
save_directory = <span class="hljs-string">&quot;a_local_path&quot;</span>
model.save_pretrained(save_directory)
tokenizer.save_pretrained(save_directory)<!-- HTML_TAG_END --></pre></div>
<p>By default, <code>OVModelForXxx</code> support dynamic shapes, enabling inputs of every shapes. To decrease latency, static shapes can be enabled by giving the desired inputs shapes.</p>

	<div class="code-block relative"><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5   text-gray-600 " title="code excerpt" type="button"><svg class="" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg>
	<div class="absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0"><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent; "></div>
	Copied</div></button></div>
	<pre><!-- HTML_TAG_START --><span class="hljs-comment"># Fix the batch size to 1 and the sequence length to 9</span>
model.reshape(<span class="hljs-number">1</span>, <span class="hljs-number">9</span>)<!-- HTML_TAG_END --></pre></div>
<p>Currently, OpenVINO only supports static shapes for inference on GPU. FP16 precision can also be enabled in order to further decrease latency.</p>

	<div class="code-block relative"><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5   text-gray-600 " title="code excerpt" type="button"><svg class="" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg>
	<div class="absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0"><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent; "></div>
	Copied</div></button></div>
	<pre><!-- HTML_TAG_START --><span class="hljs-comment"># Fix the batch size to 1 and the sequence length to 9</span>
model.reshape(<span class="hljs-number">1</span>, <span class="hljs-number">9</span>)
<span class="hljs-comment"># Enable FP16 precision</span>
model.half()
model.to(<span class="hljs-string">&quot;GPU&quot;</span>)<!-- HTML_TAG_END --></pre></div>
<p>When fixing the shapes with the <code>reshape()</code> method, inference cannot be performed with an input of a different shape. When instantiating your pipeline, you can specify the maximum total input sequence length after tokenization in order for shorter sequences to be padded and for longer sequences to be truncated.</p>

	<div class="code-block relative"><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5   text-gray-600 " title="code excerpt" type="button"><svg class="" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg>
	<div class="absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0"><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent; "></div>
	Copied</div></button></div>
	<pre><!-- HTML_TAG_START --><span class="hljs-keyword">from</span> datasets <span class="hljs-keyword">import</span> load_dataset
<span class="hljs-keyword">from</span> transformers <span class="hljs-keyword">import</span> AutoTokenizer, pipeline
<span class="hljs-keyword">from</span> evaluate <span class="hljs-keyword">import</span> evaluator
<span class="hljs-keyword">from</span> optimum.intel.openvino.modeling <span class="hljs-keyword">import</span> OVModelForQuestionAnswering

model_id = <span class="hljs-string">&quot;distilbert-base-cased-distilled-squad&quot;</span>
model = OVModelForQuestionAnswering.from_pretrained(model_id, from_transformers=<span class="hljs-literal">True</span>)
model.reshape(<span class="hljs-number">1</span>, <span class="hljs-number">384</span>)
tokenizer = AutoTokenizer.from_pretrained(model_id)
eval_dataset = load_dataset(<span class="hljs-string">&quot;squad&quot;</span>, split=<span class="hljs-string">&quot;validation&quot;</span>).select(<span class="hljs-built_in">range</span>(<span class="hljs-number">50</span>))
<span class="hljs-built_in">eval</span> = evaluator(<span class="hljs-string">&quot;question-answering&quot;</span>)
qa_pipe = pipeline(
    <span class="hljs-string">&quot;question-answering&quot;</span>,
    model=model,
    tokenizer=tokenizer,
    max_seq_len=<span class="hljs-number">384</span>,
    padding=<span class="hljs-string">&quot;max_length&quot;</span>,
    truncation=<span class="hljs-literal">True</span>,
)
metric = <span class="hljs-built_in">eval</span>.compute(model_or_pipeline=qa_pipe, data=eval_dataset, metric=<span class="hljs-string">&quot;squad&quot;</span>)<!-- HTML_TAG_END --></pre></div>
<h2 class="relative group"><a id="run-inference-for-sequencetosequence-models" class="header-link block pr-1.5 text-lg no-hover:hidden with-hover:absolute with-hover:p-1.5 with-hover:opacity-0 with-hover:group-hover:opacity-100 with-hover:right-full" href="#run-inference-for-sequencetosequence-models"><span><svg class="" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" aria-hidden="true" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 256 256"><path d="M167.594 88.393a8.001 8.001 0 0 1 0 11.314l-67.882 67.882a8 8 0 1 1-11.314-11.315l67.882-67.881a8.003 8.003 0 0 1 11.314 0zm-28.287 84.86l-28.284 28.284a40 40 0 0 1-56.567-56.567l28.284-28.284a8 8 0 0 0-11.315-11.315l-28.284 28.284a56 56 0 0 0 79.196 79.197l28.285-28.285a8 8 0 1 0-11.315-11.314zM212.852 43.14a56.002 56.002 0 0 0-79.196 0l-28.284 28.284a8 8 0 1 0 11.314 11.314l28.284-28.284a40 40 0 0 1 56.568 56.567l-28.285 28.285a8 8 0 0 0 11.315 11.314l28.284-28.284a56.065 56.065 0 0 0 0-79.196z" fill="currentColor"></path></svg></span></a>
	<span>Run inference for sequence-to-sequence models
	</span></h2>

<p>Sequence-to-sequence (Seq2Seq) models, that generate a new sequence from an input, can also be used when running inference with OpenVINO. When Seq2Seq models are exported to the OpenVINO IR, they are decomposed into two parts : the encoder and the “decoder” (which actually consists of the decoder with the language modeling head), that are later combined during inference.
To leverage the pre-computed key/values hidden-states to speed up sequential decoding, simply pass <code>use_cache=True</code> to the <code>from_pretrained()</code> method. An additional model component will be exported: the “decoder” with pre-computed key/values as one of its inputs.
This specific export comes from the fact that during the first pass, the decoder has no pre-computed key/values hidden-states, while during the rest of the generation past key/values will be used to speed up sequential decoding.
Here is an example on how you can run inference for a translation task using an MarianMT model and then export it to the OpenVINO IR:</p>

	<div class="code-block relative"><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5   text-gray-600 " title="code excerpt" type="button"><svg class="" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg>
	<div class="absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0"><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent; "></div>
	Copied</div></button></div>
	<pre><!-- HTML_TAG_START --><span class="hljs-keyword">from</span> transformers <span class="hljs-keyword">import</span> AutoTokenizer, pipeline
<span class="hljs-keyword">from</span> optimum.intel.openvino <span class="hljs-keyword">import</span> OVModelForSeq2SeqLM

model_id = <span class="hljs-string">&quot;Helsinki-NLP/opus-mt-en-fr&quot;</span>
model = OVModelForSeq2SeqLM.from_pretrained(model_id, from_transformers=<span class="hljs-literal">True</span>)
tokenizer = AutoTokenizer.from_pretrained(model_id)
translation_pipe = pipeline(<span class="hljs-string">&quot;translation_en_to_fr&quot;</span>, model=model, tokenizer=tokenizer)
text = <span class="hljs-string">&quot;He never went out without a book under his arm, and he often came back with two.&quot;</span>
result = translation_pipe(text)

<span class="hljs-comment"># Save the exported model</span>
save_directory = <span class="hljs-string">&quot;a_local_path&quot;</span>
model.save_pretrained(save_directory)
tokenizer.save_pretrained(save_directory)

[{<span class="hljs-string">&#x27;translation_text&#x27;</span>: <span class="hljs-string">&quot;Il n&#x27;est jamais sorti sans un livre sous son bras, et il est souvent revenu avec deux.&quot;</span>}]<!-- HTML_TAG_END --></pre></div>


		<script type="module" data-hydrate="1pjxa3">
		import { start } from "/docs/optimum.intel/pr_439/en/_app/start-hf-doc-builder.js";
		start({
			target: document.querySelector('[data-hydrate="1pjxa3"]').parentNode,
			paths: {"base":"/docs/optimum.intel/pr_439/en","assets":"/docs/optimum.intel/pr_439/en"},
			session: {},
			route: false,
			spa: false,
			trailing_slash: "never",
			hydrate: {
				status: 200,
				error: null,
				nodes: [
					import("/docs/optimum.intel/pr_439/en/_app/pages/__layout.svelte-hf-doc-builder.js"),
						import("/docs/optimum.intel/pr_439/en/_app/pages/inference.mdx-hf-doc-builder.js")
				],
				params: {}
			}
		});
	</script>
