title: Hogwild! Inference: Parallel LLM Generation via Concurrent Attention
summary: Large Language Models (LLMs) have demonstrated the ability to tackle
  increasingly complex tasks through advanced reasoning, long-form content
  generation, and tool use. Solving these tasks often involves long
  inference-time computations. In human problem solving, a common strategy to
  expedite work is collaboration: by dividing the problem into sub-tasks,
  exploring different strategies concurrently, etc. Recent research has shown
  that LLMs can also operate in parallel by implementing explicit cooperation
  frameworks, such as voting mechanisms or the explicit creation of independent
  sub-tasks that can be executed in parallel. However, each of these frameworks
  may not be suitable for all types of tasks, which can hinder their
  applicability. In this work, we propose a different design approach: we run LLM
  "workers" in parallel , allowing them to synchronize via a concurrently-updated
  attention cache and prompt these workers to decide how best to collaborate. Our
  approach allows the instances to come up with their own collaboration strategy
  for the problem at hand, all the while "seeing" each other's partial progress
  in the concurrent cache. We implement this approach via Hogwild! Inference: a
  parallel LLM inference engine where multiple instances of the same LLM run in
  parallel with the same attention cache, with "instant" access to each other's
  generated tokens. Hogwild! inference takes advantage of Rotary Position
  Embeddings (RoPE) to avoid recomputation while improving parallel hardware
  utilization. We find that modern reasoning-capable LLMs can perform inference
  with shared Key-Value cache out of the box, without additional fine-tuning.
paper_authors._id: 67f60df2d0df7eccaae93eb0
paper_authors.name: Gleb Rodionov
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-10T08:00:46.736Z
paper_authors._id: 67f77a13c4266c8ae9f3a299
paper_authors.avatarUrl: /avatars/37059bcfdc20bae69172354985ee57aa.svg
paper_authors.fullname: Gleb
paper_authors.type: user
paper_authors.user: eqimp
paper_authors._id: 67f60df2d0df7eccaae93eb1
paper_authors.name: Roman Garipov
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-09T14:36:56.157Z
paper_authors._id: 6261af8040e04009e813a43d
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6261af8040e04009e813a43d/Cc4I98uaePHE-YglLodlZ.jpeg
paper_authors.fullname: Roman Garipov
paper_authors.type: user
paper_authors.user: garipovroma
paper_authors._id: 67f60df2d0df7eccaae93eb2
paper_authors.name: Alina Shutova
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-09T16:31:48.423Z
paper_authors._id: 6492b61da34fecfae1175ae6
paper_authors.avatarUrl: /avatars/08ee7ffbba95f9363a7f1cd19c5a72db.svg
paper_authors.fullname: Alina Shutova
paper_authors.type: user
paper_authors.user: Alshutova
paper_authors._id: 67f60df2d0df7eccaae93eb3
paper_authors.name: George Yakushev
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-09T16:50:42.359Z
paper_authors._id: 655f8492e3dda95b340822ae
paper_authors.avatarUrl: /avatars/69e2bed8045faf35105e15f228db0567.svg
paper_authors.fullname: George
paper_authors.type: user
paper_authors.user: MrDarkTesla
paper_authors._id: 67f60df2d0df7eccaae93eb4
paper_authors.name: Vage Egiazarian
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-09T16:32:06.480Z
paper_authors._id: 64b6c0862662b49f164bfdb1
paper_authors.avatarUrl: /avatars/5a89a3eb7923d9579ad74ff77d288d37.svg
paper_authors.fullname: Vage Egiazarian
paper_authors.type: user
paper_authors.user: Vahe527887
paper_authors._id: 67f60df2d0df7eccaae93eb5
paper_authors.name: Anton Sinitsin
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-09T16:32:12.347Z
paper_authors._id: 67cd69a25221b49814460129
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/dCjdggjVowuuI-cIM8pLa.png
paper_authors.fullname: Anton Sinitsin
paper_authors.type: user
paper_authors.user: grievousxtinkt
paper_authors._id: 67f60df2d0df7eccaae93eb6
paper_authors.name: Denis Kuznedelev
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-09T16:32:18.990Z
paper_authors._id: 629cf0475a13ba8233dd18c9
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1654452258405-noauth.jpeg
paper_authors.fullname: Denis Kuznedelev
paper_authors.type: user
paper_authors.user: SpiridonSunRotator
paper_authors._id: 67f60df2d0df7eccaae93eb7
paper_authors.name: Dan Alistarh
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-09T16:32:25.593Z
paper_authors._id: 64d100c5d8d0927372e3d4c0
paper_authors.avatarUrl: /avatars/91d9e4f1dab25b70d901783cdfcd2fd1.svg
paper_authors.fullname: Dan Alistarh
paper_authors.type: user
paper_authors.user: dalistarh
paper_title: Hogwild! Inference: Parallel LLM Generation via Concurrent Attention
paper_summary: Large Language Models (LLMs) have demonstrated the ability to tackle
  increasingly complex tasks through advanced reasoning, long-form content
  generation, and tool use. Solving these tasks often involves long
  inference-time computations. In human problem solving, a common strategy to
  expedite work is collaboration: by dividing the problem into sub-tasks,
  exploring different strategies concurrently, etc. Recent research has shown
  that LLMs can also operate in parallel by implementing explicit cooperation
  frameworks, such as voting mechanisms or the explicit creation of independent
  sub-tasks that can be executed in parallel. However, each of these frameworks
  may not be suitable for all types of tasks, which can hinder their
  applicability. In this work, we propose a different design approach: we run LLM
  "workers" in parallel , allowing them to synchronize via a concurrently-updated
  attention cache and prompt these workers to decide how best to collaborate. Our
  approach allows the instances to come up with their own collaboration strategy
  for the problem at hand, all the while "seeing" each other's partial progress
  in the concurrent cache. We implement this approach via Hogwild! Inference: a
  parallel LLM inference engine where multiple instances of the same LLM run in
  parallel with the same attention cache, with "instant" access to each other's
  generated tokens. Hogwild! inference takes advantage of Rotary Position
  Embeddings (RoPE) to avoid recomputation while improving parallel hardware
  utilization. We find that modern reasoning-capable LLMs can perform inference
  with shared Key-Value cache out of the box, without additional fine-tuning.
paper_ai_summary: A parallel inference engine for LLMs allows concurrent task collaboration through a shared attention cache, improving efficiency without additional fine-tuning.
paper_ai_keywords: Large Language Models (LLMs) parallel inference cooperation frameworks voting mechanisms independent sub-tasks Hogwild! Inference attention cache generated tokens Rotary Position Embeddings (RoPE) Key-Value cache