<meta charset="utf-8" /><meta http-equiv="content-security-policy" content=""><meta name="hf:doc:metadata" content="{&quot;local&quot;:&quot;sequencetosequence-models&quot;,&quot;title&quot;:&quot;Sequence-to-sequence models&quot;}" data-svelte="svelte-1phssyn">
	<link rel="modulepreload" href="/docs/course/pr_92/en/_app/assets/pages/__layout.svelte-9b9c9c55.css">
	<link rel="modulepreload" href="/docs/course/pr_92/en/_app/start-df6144de.js">
	<link rel="modulepreload" href="/docs/course/pr_92/en/_app/chunks/vendor-1e8b365d.js">
	<link rel="modulepreload" href="/docs/course/pr_92/en/_app/chunks/paths-4b3c6e7e.js">
	<link rel="modulepreload" href="/docs/course/pr_92/en/_app/pages/__layout.svelte-05c2fb0c.js">
	<link rel="modulepreload" href="/docs/course/pr_92/en/_app/pages/chapter1/7.mdx-50c5a908.js">
	<link rel="modulepreload" href="/docs/course/pr_92/en/_app/chunks/Youtube-c2a8cc39.js">
	<link rel="modulepreload" href="/docs/course/pr_92/en/_app/chunks/IconCopyLink-483c28ba.js"> 





<h1 class="relative group"><a id="sequencetosequence-models" class="header-link block pr-1.5 text-lg no-hover:hidden with-hover:absolute with-hover:p-1.5 with-hover:opacity-0 with-hover:group-hover:opacity-100 with-hover:right-full" href="#sequencetosequence-models"><span><svg class="" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" aria-hidden="true" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 256 256"><path d="M167.594 88.393a8.001 8.001 0 0 1 0 11.314l-67.882 67.882a8 8 0 1 1-11.314-11.315l67.882-67.881a8.003 8.003 0 0 1 11.314 0zm-28.287 84.86l-28.284 28.284a40 40 0 0 1-56.567-56.567l28.284-28.284a8 8 0 0 0-11.315-11.315l-28.284 28.284a56 56 0 0 0 79.196 79.197l28.285-28.285a8 8 0 1 0-11.315-11.314zM212.852 43.14a56.002 56.002 0 0 0-79.196 0l-28.284 28.284a8 8 0 1 0 11.314 11.314l28.284-28.284a40 40 0 0 1 56.568 56.567l-28.285 28.285a8 8 0 0 0 11.315 11.314l28.284-28.284a56.065 56.065 0 0 0 0-79.196z" fill="currentColor"></path></svg></span></a>
	<span>Sequence-to-sequence models
	</span></h1>

<iframe class="w-full xl:w-4/6 h-80" src="https://www.youtube-nocookie.com/embed/0_4KEb08xrE" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe>
<p>Encoder-decoder models (also called <em>sequence-to-sequence models</em>) use both parts of the Transformer architecture. At each stage, the attention layers of the encoder can access all the words in the initial sentence, whereas the attention layers of the decoder can only access the words positioned before a given word in the input.</p>
<p>The pretraining of these models can be done using the objectives of encoder or decoder models, but usually involves something a bit more complex. For instance, <a href="https://huggingface.co/t5-base" rel="nofollow">T5</a> is pretrained by replacing random spans of text (that can contain several words) with a single mask special word, and the objective is then to predict the text that this mask word replaces.</p>
<p>Sequence-to-sequence models are best suited for tasks revolving around generating new sentences depending on a given input, such as summarization, translation, or generative question answering.</p>
<p>Representatives of this family of models include:</p>
<ul><li><a href="https://huggingface.co/transformers/model_doc/bart.html" rel="nofollow">BART</a></li>
<li><a href="https://huggingface.co/transformers/model_doc/mbart.html" rel="nofollow">mBART</a></li>
<li><a href="https://huggingface.co/transformers/model_doc/marian.html" rel="nofollow">Marian</a></li>
<li><a href="https://huggingface.co/transformers/model_doc/t5.html" rel="nofollow">T5</a></li></ul>


		<script type="module" data-hydrate="f5o3gz">
		import { start } from "/docs/course/pr_92/en/_app/start-df6144de.js";
		start({
			target: document.querySelector('[data-hydrate="f5o3gz"]').parentNode,
			paths: {"base":"/docs/course/pr_92/en","assets":"/docs/course/pr_92/en"},
			session: {},
			route: false,
			spa: false,
			trailing_slash: "never",
			hydrate: {
				status: 200,
				error: null,
				nodes: [
					import("/docs/course/pr_92/en/_app/pages/__layout.svelte-05c2fb0c.js"),
						import("/docs/course/pr_92/en/_app/pages/chapter1/7.mdx-50c5a908.js")
				],
				params: {}
			}
		});
	</script>
