title: LPZero: Language Model Zero-cost Proxy Search from Zero
summary: In spite of the outstanding performance, Neural Architecture Search (NAS) is
  criticized for massive computation. Recently, Zero-shot NAS has emerged as a
  promising approach by exploiting Zero-cost (ZC) proxies, which markedly reduce
  computational demands. Despite this, existing ZC proxies heavily rely on expert
  knowledge and incur significant trial-and-error costs. Particularly in NLP
  tasks, most existing ZC proxies fail to surpass the performance of the naive
  baseline. To address these challenges, we introduce a novel framework,
  LPZero, which is the first to automatically design ZC proxies for
  various tasks, achieving higher ranking consistency than human-designed
  proxies. Specifically, we model the ZC proxy as a symbolic equation and
  incorporate a unified proxy search space that encompasses existing ZC proxies,
  which are composed of a predefined set of mathematical symbols. To
  heuristically search for the best ZC proxy, LPZero incorporates genetic
  programming to find the optimal symbolic composition. We propose a
  Rule-based Pruning Strategy (RPS), which preemptively eliminates
  unpromising proxies, thereby mitigating the risk of proxy degradation.
  Extensive experiments on FlexiBERT, GPT-2, and LLaMA-7B demonstrate LPZero's
  superior ranking ability and performance on downstream tasks compared to
  current approaches.
paper_authors._id: 67092a3453dcd91dc5585b51
paper_authors.name: Peijie Dong
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-06-02T07:57:07.734Z
paper_authors._id: 6395f845aec00abff778ad31
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6395f845aec00abff778ad31/bZkAlchSvqER1HgBKmcHI.jpeg
paper_authors.fullname: PeijieDong
paper_authors.type: user
paper_authors.user: pprp
paper_authors._id: 67092a3453dcd91dc5585b52
paper_authors.name: Lujun Li
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2024-10-11T13:38:26.191Z
paper_authors._id: 65ff0330d157381f164e009d
paper_authors.avatarUrl: /avatars/da502aa33a41f3076cea3def69006db6.svg
paper_authors.fullname: luking66
paper_authors.type: user
paper_authors.user: luking66
paper_authors._id: 67092a3453dcd91dc5585b53
paper_authors.name: Xiang Liu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-25T09:31:20.929Z
paper_authors._id: 63024676056ec3a2a8714b24
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1661093436322-noauth.jpeg
paper_authors.fullname: Xiang Liu
paper_authors.type: user
paper_authors.user: Dominic789654
paper_authors._id: 67092a3453dcd91dc5585b54
paper_authors.name: Zhenheng Tang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-14T12:42:13.582Z
paper_authors._id: 66a4a319a1711696948b045c
paper_authors.avatarUrl: /avatars/1d92d57a949332cb8227697b9a0c2f39.svg
paper_authors.fullname: Zhenheng Tang
paper_authors.type: user
paper_authors.user: coolzhtang
paper_authors._id: 67092a3453dcd91dc5585b55
paper_authors.name: Xuebo Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-14T12:42:19.289Z
paper_authors._id: 652fbf9e75f5f7a84e10cb2e
paper_authors.avatarUrl: /avatars/73e7af465dbb57826a52e49c3e72f55f.svg
paper_authors.fullname: Xuebo Liu
paper_authors.type: user
paper_authors.user: SunbowLiu
paper_authors._id: 67092a3453dcd91dc5585b56
paper_authors.name: Qiang Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-14T12:42:26.113Z
paper_authors._id: 6625e033f5c285535cc5d113
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/aNWyzeYR3xH1fdqljSTfv.jpeg
paper_authors.fullname: qang_wang
paper_authors.type: user
paper_authors.user: QiangWang0
paper_authors._id: 67092a3453dcd91dc5585b57
paper_authors.name: Xiaowen Chu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-14T12:42:32.107Z
paper_authors._id: 6676935fcd0b89a0115174b0
paper_authors.avatarUrl: /avatars/4caca1b672d29e787814f9a30bf20bcc.svg
paper_authors.fullname: Xiaowen Chu
paper_authors.type: user
paper_authors.user: wenxinsiju
paper_title: LPZero: Language Model Zero-cost Proxy Search from Zero
paper_summary: In spite of the outstanding performance, Neural Architecture Search (NAS) is
  criticized for massive computation. Recently, Zero-shot NAS has emerged as a
  promising approach by exploiting Zero-cost (ZC) proxies, which markedly reduce
  computational demands. Despite this, existing ZC proxies heavily rely on expert
  knowledge and incur significant trial-and-error costs. Particularly in NLP
  tasks, most existing ZC proxies fail to surpass the performance of the naive
  baseline. To address these challenges, we introduce a novel framework,
  LPZero, which is the first to automatically design ZC proxies for
  various tasks, achieving higher ranking consistency than human-designed
  proxies. Specifically, we model the ZC proxy as a symbolic equation and
  incorporate a unified proxy search space that encompasses existing ZC proxies,
  which are composed of a predefined set of mathematical symbols. To
  heuristically search for the best ZC proxy, LPZero incorporates genetic
  programming to find the optimal symbolic composition. We propose a
  Rule-based Pruning Strategy (RPS), which preemptively eliminates
  unpromising proxies, thereby mitigating the risk of proxy degradation.
  Extensive experiments on FlexiBERT, GPT-2, and LLaMA-7B demonstrate LPZero's
  superior ranking ability and performance on downstream tasks compared to
  current approaches.
paper_ai_summary: LPZero, a novel framework, automates the design of Zero-cost proxies using genetic programming and achieves better performance than human-designed proxies on various NLP tasks.
paper_ai_keywords: Neural Architecture Search Zero-shot NAS Zero-cost proxies symbolic equation proxy search space genetic programming Rule-based Pruning Strategy FlexiBERT GPT-2 LLaMA-7B