La robotique humanoïde et l'IA incarnée s'apprête à changer le monde
Il est alors possible de simuler les actions du robot à des vitesses des millions de fois supérieur à la réalité dans un monde digital, et de ...
Attention is all you need
C’est alors qu’en 2018, des chercheurs de Google publièrent un article de recherche qui allait tout changer : <anchor>Attention is all you need</anchor>. Cet article présente une nouvelle architecture d’IA appelée le Transformer, qui présente de nombreux atouts face aux modèles existants, dont une haute capacité de parallélisation. Ce modèle est tellement efficace qu’il est a l’origine de tous les outils d’IA générative que l’on connait aujourd’hui, dont GPT4, pour Generative Pre-trained Transformer v4.
[1706.03762] Attention Is All You Need
\[1706.03762\] Attention Is All You Need![close this message](/static/browse/0.3.4/images/icons/close-slider.png)

![arXiv Accessibility Forum 2024](/static/browse/0.3.4/images/events/arxiv-forum-logotype-title.png)

[Skip to main content](#content)

[![Cornell University](/static/browse/0.3.4/images/icons/cu/cornell-reduced-white-SMALL.svg)](https://www.cornell.edu/)

This week: the arXiv Accessibility Forum

[Forum Schedule](https://accessibility2024.arxiv.org/schedule)

We gratefully acknowledge support from the Simons Foundation, [member institutions](https://info.arxiv.org/about/ourmembers.html), and all contributors. [Donate](https://info.arxiv.org/about/donate.html)

[![arxiv logo](/static/browse/0.3.4/images/arxiv-logo-one-color-white.svg)](/) > [cs](/list/cs/recent) > arXiv:1706.03762


[Help](https://info.arxiv.org/help) \| [Advanced Search](https://arxiv.org/search/advanced)

All fieldsTitleAuthorAbstractCommentsJournal referenceACM classificationMSC classificationReport numberarXiv identifierDOIORCIDarXiv author IDHelp pagesFull text

Search

[![arXiv logo](/static/browse/0.3.4/images/arxiv-logomark-small-white.svg)](https://arxiv.org/)

[![Cornell University Logo](/static/browse/0.3.4/images/icons/cu/cornell-reduced-white-SMALL.svg)](https://www.cornell.edu/)

open search

GO

open navigation menu

## quick links

- [Login](https://arxiv.org/login)
- [Help Pages](https://info.arxiv.org/help)
- [About](https://info.arxiv.org/about)

# Computer Science > Computation and Language

**arXiv:1706.03762** (cs)


\[Submitted on 12 Jun 2017 ( [v1](https://arxiv.org/abs/1706.03762v1)), last revised 2 Aug 2023 (this version, v7)\]

# Title:Attention Is All You Need

Authors: [Ashish Vaswani](https://arxiv.org/search/cs?searchtype=author&query=Vaswani,+A), [Noam Shazeer](https://arxiv.org/search/cs?searchtype=author&query=Shazeer,+N), [Niki Parmar](https://arxiv.org/search/cs?searchtype=author&query=Parmar,+N), [Jakob Uszkoreit](https://arxiv.org/search/cs?searchtype=author&query=Uszkoreit,+J), [Llion Jones](https://arxiv.org/search/cs?searchtype=author&query=Jones,+L), [Aidan N. Gomez](https://arxiv.org/search/cs?searchtype=author&query=Gomez,+A+N), [Lukasz Kaiser](https://arxiv.org/search/cs?searchtype=author&query=Kaiser,+L), [Illia Polosukhin](https://arxiv.org/search/cs?searchtype=author&query=Polosukhin,+I)

View a PDF of the paper titled Attention Is All You Need, by Ashish Vaswani and 7 other authors

[View PDF](/pdf/1706.03762) [HTML (experimental)](https://arxiv.org/html/1706.03762v7)

> Abstract:The dominant sequence transduction models are based on complex recurrent or convolutional neural networks in an encoder-decoder configuration. The best performing models also connect the encoder and decoder through an attention mechanism. We propose a new simple network architecture, the Transformer, based solely on attention mechanisms, dispensing with recurrence and convolutions entirely. Experiments on two machine translation tasks show these models to be superior in quality while being more parallelizable and requiring significantly less time to train. Our model achieves 28.4 BLEU on the WMT 2014 English-to-German translation task, improving over the existing best results, including ensembles by over 2 BLEU. On the WMT 2014 English-to-French translation task, our model establishes a new single-model state-of-the-art BLEU score of 41.8 after training for 3.5 days on eight GPUs, a small fraction of the training costs of the best models from the literature. We show that the Transformer generalizes well to other tasks by applying it successfully to English constituency parsing both with large and limited training data.

Comments:15 pages, 5 figuresSubjects:Computation and Language (cs.CL); Machine Learning (cs.LG)Cite as:[arXiv:1706.03762](https://arxiv.org/abs/1706.03762) \[cs.CL\](or [arXiv:1706.03762v7](https://arxiv.org/abs/1706.03762v7) \[cs.CL\] for this version)
[https://doi.org/10.48550/arXiv.1706.03762](https://doi.org/10.48550/arXiv.1706.03762)

Focus to learn more

arXiv-issued DOI via DataCite

## Submission history

From: Llion Jones \[ [view email](/show-email/f53b7360/1706.03762)\]

**[\[v1\]](/abs/1706.03762v1)**
Mon, 12 Jun 2017 17:57:34 UTC (1,102 KB)

**[\[v2\]](/abs/1706.03762v2)**
Mon, 19 Jun 2017 16:49:45 UTC (1,125 KB)

**[\[v3\]](/abs/1706.03762v3)**
Tue, 20 Jun 2017 05:20:02 UTC (1,125 KB)

**[\[v4\]](/abs/1706.03762v4)**
Fri, 30 Jun 2017 17:29:30 UTC (1,124 KB)

**[\[v5\]](/abs/1706.03762v5)**
Wed, 6 Dec 2017 03:30:32 UTC (1,124 KB)

**[\[v6\]](/abs/1706.03762v6)**
Mon, 24 Jul 2023 00:48:54 UTC (1,124 KB)

**\[v7\]**
Wed, 2 Aug 2023 00:41:18 UTC (1,124 KB)

Full-text links:

## Access Paper:

View a PDF of the paper titled Attention Is All You Need, by Ashish Vaswani and 7 other authors

- [View PDF](/pdf/1706.03762)
- [HTML (experimental)](https://arxiv.org/html/1706.03762v7)
- [TeX Source](/src/1706.03762)
- [Other Formats](/format/1706.03762)

[view license](http://arxiv.org/licenses/nonexclusive-distrib/1.0/ "Rights to this article")

Current browse context:

cs.CL

[< prev](/prevnext?id=1706.03762&function=prev&context=cs.CL "previous in cs.CL (accesskey p)")  \|  [next >](/prevnext?id=1706.03762&function=next&context=cs.CL "next in cs.CL (accesskey n)")

[new](/list/cs.CL/new) \| [recent](/list/cs.CL/recent) \| [2017-06](/list/cs.CL/2017-06)

Change to browse by:


[cs](/abs/1706.03762?context=cs)

[cs.LG](/abs/1706.03762?context=cs.LG)

### References & Citations

- [NASA ADS](https://ui.adsabs.harvard.edu/abs/arXiv:1706.03762)
- [Google Scholar](https://scholar.google.com/scholar_lookup?arxiv_id=1706.03762)
- [Semantic Scholar](https://api.semanticscholar.org/arXiv:1706.03762)

### [119 blog links](/tb/1706.03762)

( [what is this?](https://info.arxiv.org/help/trackback.html))


### [DBLP](https://dblp.uni-trier.de) \- CS Bibliography

[listing](https://dblp.uni-trier.de/db/journals/corr/corr1706.html#VaswaniSPUJGKP17 "listing on DBLP") \| [bibtex](https://dblp.uni-trier.de/rec/bibtex/journals/corr/VaswaniSPUJGKP17 "DBLP bibtex record")

[Ashish Vaswani](https://dblp.uni-trier.de/search/author?author=Ashish%20Vaswani "DBLP author search")

[Noam Shazeer](https://dblp.uni-trier.de/search/author?author=Noam%20Shazeer "DBLP author search")

[Niki Parmar](https://dblp.uni-trier.de/search/author?author=Niki%20Parmar "DBLP author search")

[Jakob Uszkoreit](https://dblp.uni-trier.de/search/author?author=Jakob%20Uszkoreit "DBLP author search")

[Llion Jones](https://dblp.uni-trier.de/search/author?author=Llion%20Jones "DBLP author search")

…

[a](/static/browse/0.3.4/css/cite.css) export BibTeX citationLoading...

## BibTeX formatted citation

×

Data provided by:

### Bookmark

[![BibSonomy logo](/static/browse/0.3.4/images/icons/social/bibsonomy.png)](http://www.bibsonomy.org/BibtexHandler?requTask=upload&url=https://arxiv.org/abs/1706.03762&description=Attention Is All You Need "Bookmark on BibSonomy") [![Reddit logo](/static/browse/0.3.4/images/icons/social/reddit.png)](https://reddit.com/submit?url=https://arxiv.org/abs/1706.03762&title=Attention Is All You Need "Bookmark on Reddit")

Bibliographic Tools

# Bibliographic and Citation Tools

Bibliographic Explorer Toggle

Bibliographic Explorer _( [What is the Explorer?](https://info.arxiv.org/labs/showcase.html#arxiv-bibliographic-explorer))_

Litmaps Toggle

Litmaps _( [What is Litmaps?](https://www.litmaps.co/))_

scite.ai Toggle

scite Smart Citations _( [What are Smart Citations?](https://www.scite.ai/))_

Code, Data, Media

# Code, Data and Media Associated with this Article

Links to Code Toggle

CatalyzeX Code Finder for Papers _( [What is CatalyzeX?](https://www.catalyzex.com))_

DagsHub Toggle

DagsHub _( [What is DagsHub?](https://dagshub.com/))_

GotitPub Toggle

Gotit.pub _( [What is GotitPub?](http://gotit.pub/faq))_

Links to Code Toggle

Papers with Code _( [What is Papers with Code?](https://paperswithcode.com/))_

ScienceCast Toggle

ScienceCast _( [What is ScienceCast?](https://sciencecast.org/welcome))_

Demos

# Demos

Replicate Toggle

Replicate _( [What is Replicate?](https://replicate.com/docs/arxiv/about))_

Spaces Toggle

Hugging Face Spaces _( [What is Spaces?](https://huggingface.co/docs/hub/spaces))_

Spaces Toggle

TXYZ.AI _( [What is TXYZ.AI?](https://txyz.ai))_

Related Papers

# Recommenders and Search Tools

Link to Influence Flower

Influence Flower _( [What are Influence Flowers?](https://influencemap.cmlab.dev/))_

Connected Papers Toggle

Connected Papers _( [What is Connected Papers?](https://www.connectedpapers.com/about))_

Core recommender toggle

CORE Recommender _( [What is CORE?](https://core.ac.uk/services/recommender))_

- Author
- Venue
- Institution
- Topic

About arXivLabs


# arXivLabs: experimental projects with community collaborators

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? [**Learn more about arXivLabs**](https://info.arxiv.org/labs/index.html).

[Which authors of this paper are endorsers?](/auth/show-endorsers/1706.03762) \|
[Disable MathJax](javascript:setMathjaxCookie()) ( [What is MathJax?](https://info.arxiv.org/help/mathjax.html))


- [About](https://info.arxiv.org/about)
- [Help](https://info.arxiv.org/help)

- contact arXivClick here to contact arXiv [Contact](https://info.arxiv.org/help/contact.html)
- subscribe to arXiv mailingsClick here to subscribe [Subscribe](https://info.arxiv.org/help/subscribe)

- [Copyright](https://info.arxiv.org/help/license/index.html)
- [Privacy Policy](https://info.arxiv.org/help/policies/privacy_policy.html)

- [Web Accessibility Assistance](https://info.arxiv.org/help/web_accessibility.html)
- [arXiv Operational Status](https://status.arxiv.org)


Get status notifications via
[email](https://subscribe.sorryapp.com/24846f03/email/new)
or [slack](https://subscribe.sorryapp.com/24846f03/slack/new)