title: PUMA: Secure Inference of LLaMA-7B in Five Minutes
summary: With ChatGPT as a representative, tons of companies have began to provide
  services based on large Transformers models. However, using such a service
  inevitably leak users' prompts to the model provider. Previous studies have
  studied secure inference for Transformer models using secure multiparty
  computation (MPC), where model parameters and clients' prompts are kept secret.
  Despite this, these frameworks are still limited in terms of model performance,
  efficiency, and deployment. To address these limitations, we propose framework
  PUMA to enable fast and secure Transformer model inference. Our framework
  designs high quality approximations for expensive functions, such as GeLU and
  Softmax, which significantly reduce the cost of secure inference while
  preserving the model performance. Additionally, we design secure Embedding and
  LayerNorm procedures that faithfully implement the desired functionality
  without undermining the Transformer architecture. PUMA is about 2x faster than
  the state-of-the-art MPC framework MPCFORMER(ICLR 2023) and has similar
  accuracy as plaintext models without fine-tuning (which the previous works
  failed to achieve).
    One more thing, PUMA can evaluate LLaMA-7B in around 5 minutes to generate 1
  token. To our best knowledge, this is the first time that a model with such a
  parameter size is able to be evaluated under MPC. PUMA has been open-sourced in
  the Github repository of SecretFlow-SPU.
paper_authors._id: 64bf315b4561d0aca2d5112d
paper_authors.name: Ye Dong
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-02-15T08:29:17.695Z
paper_authors._id: 64ded395931a8e1c2c7bce03
paper_authors.avatarUrl: /avatars/f70c6644b96741650fea587e2654b9c2.svg
paper_authors.fullname: Ye Dong
paper_authors.type: user
paper_authors.user: Ye000
paper_authors._id: 64bf315b4561d0aca2d5112e
paper_authors.name: Wen-jie Lu
paper_authors._id: 64bf315b4561d0aca2d5112f
paper_authors.name: Yancheng Zheng
paper_authors._id: 64bf315b4561d0aca2d51130
paper_authors.name: Haoqi Wu
paper_authors._id: 64bf315b4561d0aca2d51131
paper_authors.name: Derun Zhao
paper_authors._id: 64bf315b4561d0aca2d51132
paper_authors.name: Jin Tan
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-08-02T20:30:31.905Z
paper_authors._id: 64843d82dcece54afd738218
paper_authors.avatarUrl: /avatars/c0b2b2eeafe8e8b58ebea4e5db03c454.svg
paper_authors.fullname: Jin Tan
paper_authors.type: user
paper_authors.user: jint001
paper_authors._id: 64bf315b4561d0aca2d51133
paper_authors.name: Zhicong Huang
paper_authors._id: 64bf315b4561d0aca2d51134
paper_authors.name: Cheng Hong
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-07-25T10:41:41.835Z
paper_authors._id: 64bf6aa501f1983a86b7f990
paper_authors.avatarUrl: /avatars/1d5514a37571aae547d071d4655e5957.svg
paper_authors.fullname: Cheng Hong
paper_authors.type: user
paper_authors.user: vincehong
paper_authors._id: 64bf315b4561d0aca2d51135
paper_authors.name: Tao Wei
paper_authors._id: 64bf315b4561d0aca2d51136
paper_authors.name: Wenguang Cheng
paper_title: PUMA: Secure Inference of LLaMA-7B in Five Minutes
paper_summary: With ChatGPT as a representative, tons of companies have began to provide
  services based on large Transformers models. However, using such a service
  inevitably leak users' prompts to the model provider. Previous studies have
  studied secure inference for Transformer models using secure multiparty
  computation (MPC), where model parameters and clients' prompts are kept secret.
  Despite this, these frameworks are still limited in terms of model performance,
  efficiency, and deployment. To address these limitations, we propose framework
  PUMA to enable fast and secure Transformer model inference. Our framework
  designs high quality approximations for expensive functions, such as GeLU and
  Softmax, which significantly reduce the cost of secure inference while
  preserving the model performance. Additionally, we design secure Embedding and
  LayerNorm procedures that faithfully implement the desired functionality
  without undermining the Transformer architecture. PUMA is about 2x faster than
  the state-of-the-art MPC framework MPCFORMER(ICLR 2023) and has similar
  accuracy as plaintext models without fine-tuning (which the previous works
  failed to achieve).
    One more thing, PUMA can evaluate LLaMA-7B in around 5 minutes to generate 1
  token. To our best knowledge, this is the first time that a model with such a
  parameter size is able to be evaluated under MPC. PUMA has been open-sourced in
  the Github repository of SecretFlow-SPU.
paper_ai_summary: PUMA framework accelerates secure Transformer model inference using high-quality approximations and secure procedures, achieving faster performance and maintaining accuracy compared to existing methods.
paper_ai_keywords: MPC Transformer models secure inference GeLU Softmax Embedding LayerNorm PUMA MPCFORMER LLaMA-7B