title: Large Language Models for Compiler Optimization
summary: We explore the novel application of Large Language Models to code
  optimization. We present a 7B-parameter transformer model trained from scratch
  to optimize LLVM assembly for code size. The model takes as input unoptimized
  assembly and outputs a list of compiler options to best optimize the program.
  Crucially, during training, we ask the model to predict the instruction counts
  before and after optimization, and the optimized code itself. These auxiliary
  learning tasks significantly improve the optimization performance of the model
  and improve the model's depth of understanding.
    We evaluate on a large suite of test programs. Our approach achieves a 3.0%
  improvement in reducing instruction counts over the compiler, outperforming two
  state-of-the-art baselines that require thousands of compilations. Furthermore,
  the model shows surprisingly strong code reasoning abilities, generating
  compilable code 91% of the time and perfectly emulating the output of the
  compiler 70% of the time.
paper_authors._id: 65026e90cccc7b28a3660451
paper_authors.name: Chris Cummins
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-14T07:33:07.487Z
paper_authors._id: 63d32301c20657d2434d2fcb
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1674781395584-noauth.jpeg
paper_authors.fullname: Chris Cummins
paper_authors.type: user
paper_authors.user: ChrisCummins
paper_authors._id: 65026e90cccc7b28a3660452
paper_authors.name: Volker Seeker
paper_authors._id: 65026e90cccc7b28a3660453
paper_authors.name: Dejan Grubisic
paper_authors._id: 65026e90cccc7b28a3660454
paper_authors.name: Mostafa Elhoushi
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-14T07:33:31.367Z
paper_authors._id: 63c9725ebedad7e2bf160bdc
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/63c9725ebedad7e2bf160bdc/wzPuyhOXCYBNGwZDshbnL.jpeg
paper_authors.fullname: Mostafa Elhoushi
paper_authors.type: user
paper_authors.user: melhoushi
paper_authors._id: 65026e90cccc7b28a3660455
paper_authors.name: Youwei Liang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-14T07:33:39.520Z
paper_authors._id: 64a475aa33ded98516c6a37b
paper_authors.avatarUrl: /avatars/3abf5f8c7f56cc8612ab590f1b75c1d5.svg
paper_authors.fullname: Youwei Liang
paper_authors.type: user
paper_authors.user: youweiliang
paper_authors._id: 65026e90cccc7b28a3660456
paper_authors.name: Baptiste Roziere
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-14T07:33:47.267Z
paper_authors._id: 62d063dac375d0c84255b9a1
paper_authors.avatarUrl: /avatars/de0fc34bad8c761210c0895ebfa4feba.svg
paper_authors.fullname: Baptiste Roziere
paper_authors.type: user
paper_authors.user: broz
paper_authors._id: 65026e90cccc7b28a3660457
paper_authors.name: Jonas Gehring
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-14T07:33:54.909Z
paper_authors._id: 6327890128396be9bfec7995
paper_authors.avatarUrl: /avatars/5216ebcee5c41990a57e7a29fe114d6c.svg
paper_authors.fullname: Jonas Gehring
paper_authors.type: user
paper_authors.user: jgehring
paper_authors._id: 65026e90cccc7b28a3660458
paper_authors.name: Fabian Gloeckle
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-14T07:34:02.396Z
paper_authors._id: 64eca847971abfd26fe965be
paper_authors.avatarUrl: /avatars/c53e119c6319768ffedc10442d4830ab.svg
paper_authors.fullname: Fabian Gloeckle
paper_authors.type: user
paper_authors.user: faaabian
paper_authors._id: 65026e90cccc7b28a3660459
paper_authors.name: Kim Hazelwood
paper_authors._id: 65026e90cccc7b28a366045a
paper_authors.name: Gabriel Synnaeve
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-14T07:34:19.086Z
paper_authors._id: 630eac7931970d1cd4fbacf2
paper_authors.avatarUrl: /avatars/b7ccbddfa745db854dc342be1327cd53.svg
paper_authors.fullname: Gabriel Synnaeve
paper_authors.type: user
paper_authors.user: gsynnaeve
paper_authors._id: 65026e90cccc7b28a366045b
paper_authors.name: Hugh Leather
paper_title: Large Language Models for Compiler Optimization
paper_summary: We explore the novel application of Large Language Models to code
  optimization. We present a 7B-parameter transformer model trained from scratch
  to optimize LLVM assembly for code size. The model takes as input unoptimized
  assembly and outputs a list of compiler options to best optimize the program.
  Crucially, during training, we ask the model to predict the instruction counts
  before and after optimization, and the optimized code itself. These auxiliary
  learning tasks significantly improve the optimization performance of the model
  and improve the model's depth of understanding.
    We evaluate on a large suite of test programs. Our approach achieves a 3.0%
  improvement in reducing instruction counts over the compiler, outperforming two
  state-of-the-art baselines that require thousands of compilations. Furthermore,
  the model shows surprisingly strong code reasoning abilities, generating
  compilable code 91% of the time and perfectly emulating the output of the
  compiler 70% of the time.
paper_ai_keywords: transformer model LLVM assembly compiler options instruction counts auxiliary learning tasks