title: Stealing Part of a Production Language Model
summary: We introduce the first model-stealing attack that extracts precise,
  nontrivial information from black-box production language models like OpenAI's
  ChatGPT or Google's PaLM-2. Specifically, our attack recovers the embedding
  projection layer (up to symmetries) of a transformer model, given typical API
  access. For under \20 USD, our attack extracts the entire projection matrix of
  OpenAI's Ada and Babbage language models. We thereby confirm, for the first
  time, that these black-box models have a hidden dimension of 1024 and 2048,
  respectively. We also recover the exact hidden dimension size of the
  gpt-3.5-turbo model, and estimate it would cost under 2,000 in queries to
  recover the entire projection matrix. We conclude with potential defenses and
  mitigations, and discuss the implications of possible future work that could
  extend our attack.
paper_authors._id: 65efc32c55f190f77e7b1d5b
paper_authors.name: Nicholas Carlini
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-12T12:07:28.322Z
paper_authors._id: 6303fa9ba362e7e8b51d8f2a
paper_authors.avatarUrl: /avatars/53e53c84f987989deb351dd2ae6ee558.svg
paper_authors.fullname: Nicholas Carlini
paper_authors.type: user
paper_authors.user: carlini
paper_authors._id: 65efc32c55f190f77e7b1d5c
paper_authors.name: Daniel Paleka
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-12T12:07:34.833Z
paper_authors._id: 62363277cd8e1f47c865190a
paper_authors.avatarUrl: /avatars/fb42d42dedd4dde8df4109883cbe7ea9.svg
paper_authors.fullname: Daniel Paleka
paper_authors.type: user
paper_authors.user: dpaleka
paper_authors._id: 65efc32c55f190f77e7b1d5d
paper_authors.name: Krishnamurthy Dj Dvijotham
paper_authors._id: 65efc32c55f190f77e7b1d5e
paper_authors.name: Thomas Steinke
paper_authors._id: 65efc32c55f190f77e7b1d5f
paper_authors.name: Jonathan Hayase
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-12T12:08:04.490Z
paper_authors._id: 62f68be78465979f627b94b6
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/62f68be78465979f627b94b6/AfygEy8RaanUbWBkIxbey.jpeg
paper_authors.fullname: Jonathan Hayase
paper_authors.type: user
paper_authors.user: Jhayase
paper_authors._id: 65efc32c55f190f77e7b1d60
paper_authors.name: A. Feder Cooper
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-12T12:08:10.892Z
paper_authors._id: 659dba7a93fde4195d173334
paper_authors.avatarUrl: /avatars/f7b7264a3007f7f0a85be6e3c698f432.svg
paper_authors.fullname: A. Feder Cooper
paper_authors.type: user
paper_authors.user: afcooper
paper_authors._id: 65efc32c55f190f77e7b1d61
paper_authors.name: Katherine Lee
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-12T12:08:32.515Z
paper_authors._id: 612a1e924f3c2ac50a06f950
paper_authors.avatarUrl: /avatars/d424605e0d48327082114b878b91b161.svg
paper_authors.fullname: Katherine Lee
paper_authors.type: user
paper_authors.user: katherine1ee
paper_authors._id: 65efc32c55f190f77e7b1d62
paper_authors.name: Matthew Jagielski
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-12T12:08:40.534Z
paper_authors._id: 654a4a3b6a46bad8c0f2ab80
paper_authors.avatarUrl: /avatars/08caf74525ea3e46fe58c20eb1f3d64d.svg
paper_authors.fullname: Matthew Jagielski
paper_authors.type: user
paper_authors.user: jagielski
paper_authors._id: 65efc32c55f190f77e7b1d63
paper_authors.name: Milad Nasr
paper_authors._id: 65efc32c55f190f77e7b1d64
paper_authors.name: Arthur Conmy
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-12T12:08:55.056Z
paper_authors._id: 631ecc58daa9591e522e1494
paper_authors.avatarUrl: /avatars/4f808fae966e808105e89712c97d90d2.svg
paper_authors.fullname: VConm
paper_authors.type: user
paper_authors.user: ArthurConmy
paper_authors._id: 65efc32c55f190f77e7b1d65
paper_authors.name: Eric Wallace
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-12T12:09:47.872Z
paper_authors._id: 63112d2431257261d20d5754
paper_authors.avatarUrl: /avatars/502a68c0fb2f0c6989fe2869d0a7e3f4.svg
paper_authors.fullname: Eric Wallace
paper_authors.type: user
paper_authors.user: EricWallace
paper_authors._id: 65efc32c55f190f77e7b1d66
paper_authors.name: David Rolnick
paper_authors._id: 65efc32c55f190f77e7b1d67
paper_authors.name: Florian Tramèr
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-12T12:10:02.779Z
paper_authors._id: 63568f18ba90b4ea9fe91cb5
paper_authors.avatarUrl: /avatars/3e8b3c573e20cf80d329a312bfc34728.svg
paper_authors.fullname: Florian Tramer
paper_authors.type: user
paper_authors.user: ftramer
paper_title: Stealing Part of a Production Language Model
paper_summary: We introduce the first model-stealing attack that extracts precise,
  nontrivial information from black-box production language models like OpenAI's
  ChatGPT or Google's PaLM-2. Specifically, our attack recovers the embedding
  projection layer (up to symmetries) of a transformer model, given typical API
  access. For under \20 USD, our attack extracts the entire projection matrix of
  OpenAI's Ada and Babbage language models. We thereby confirm, for the first
  time, that these black-box models have a hidden dimension of 1024 and 2048,
  respectively. We also recover the exact hidden dimension size of the
  gpt-3.5-turbo model, and estimate it would cost under 2,000 in queries to
  recover the entire projection matrix. We conclude with potential defenses and
  mitigations, and discuss the implications of possible future work that could
  extend our attack.
paper_ai_summary: A model-stealing attack is introduced that can extract detailed information such as the embedding projection layer from black-box language models with minimal cost.
paper_ai_keywords: model-stealing attack embedding projection layer transformer model API access black-box models OpenAI's Ada OpenAI's Babbage gpt-3.5-turbo hidden dimension projection matrix potential defenses mitigations