title: Beyond Surface: Probing LLaMA Across Scales and Layers
summary: This paper presents an in-depth analysis of Large Language Models (LLMs),
  focusing on LLaMA, a prominent open-source foundational model in natural
  language processing. Instead of assessing LLaMA through its generative output,
  we design multiple-choice tasks to probe its intrinsic understanding in
  high-order tasks such as reasoning and computation. We examine the model
  horizontally, comparing different sizes, and vertically, assessing different
  layers. We unveil several key and uncommon findings based on the designed
  probing tasks: (1) Horizontally, enlarging model sizes almost could not
  automatically impart additional knowledge or computational prowess. Instead, it
  can enhance reasoning abilities, especially in math problem solving, and helps
  reduce hallucinations, but only beyond certain size thresholds; (2) In vertical
  analysis, the lower layers of LLaMA lack substantial arithmetic and factual
  knowledge, showcasing logical thinking, multilingual and recognitive abilities,
  with top layers housing most computational power and real-world knowledge.
paper_authors._id: 65727d43379284b904a3190c
paper_authors.name: Nuo Chen
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-12-08T11:21:26.268Z
paper_authors._id: 64f5a9799bd32ae6c82e0bb4
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64f5a9799bd32ae6c82e0bb4/j7qf4vMJhSLLKqJp5zXig.png
paper_authors.fullname: Chen
paper_authors.type: user
paper_authors.user: Nuo97
paper_authors._id: 65727d43379284b904a3190d
paper_authors.name: Ning Wu
paper_authors._id: 65727d43379284b904a3190e
paper_authors.name: Shining Liang
paper_authors._id: 65727d43379284b904a3190f
paper_authors.name: Ming Gong
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-08T10:13:55.533Z
paper_authors._id: 6438ad8f1efe72ba4807e801
paper_authors.avatarUrl: /avatars/9f8d63175379fd26881cd2f597a0c17c.svg
paper_authors.fullname: Ming Gong
paper_authors.type: user
paper_authors.user: Veraaaa
paper_authors._id: 65727d43379284b904a31910
paper_authors.name: Linjun Shou
paper_authors._id: 65727d43379284b904a31911
paper_authors.name: Dongmei Zhang
paper_authors._id: 65727d43379284b904a31912
paper_authors.name: Jia Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-08T10:14:22.234Z
paper_authors._id: 64ad4f24f5d40283c6307c1e
paper_authors.avatarUrl: /avatars/0175716395e34d546282d1be4d3a5936.svg
paper_authors.fullname: Ji-a Lee
paper_authors.type: user
paper_authors.user: jialee0122
paper_title: Beyond Surface: Probing LLaMA Across Scales and Layers
paper_summary: This paper presents an in-depth analysis of Large Language Models (LLMs),
  focusing on LLaMA, a prominent open-source foundational model in natural
  language processing. Instead of assessing LLaMA through its generative output,
  we design multiple-choice tasks to probe its intrinsic understanding in
  high-order tasks such as reasoning and computation. We examine the model
  horizontally, comparing different sizes, and vertically, assessing different
  layers. We unveil several key and uncommon findings based on the designed
  probing tasks: (1) Horizontally, enlarging model sizes almost could not
  automatically impart additional knowledge or computational prowess. Instead, it
  can enhance reasoning abilities, especially in math problem solving, and helps
  reduce hallucinations, but only beyond certain size thresholds; (2) In vertical
  analysis, the lower layers of LLaMA lack substantial arithmetic and factual
  knowledge, showcasing logical thinking, multilingual and recognitive abilities,
  with top layers housing most computational power and real-world knowledge.
paper_ai_keywords: Large Language Models (LLMs) LLaMA natural language processing multiple-choice tasks high-order tasks reasoning computation pairwise tasks model sizes layers hallucinations lower layers arithmetic factual knowledge logical thinking multilingual recognitive abilities top layers step-by-step breakdown justifying the extracted terms: