title: Humanity's Last Exam
summary: Benchmarks are important tools for tracking the rapid advancements in large
  language model (LLM) capabilities. However, benchmarks are not keeping pace in
  difficulty: LLMs now achieve over 90\% accuracy on popular benchmarks like
  MMLU, limiting informed measurement of state-of-the-art LLM capabilities. In
  response, we introduce Humanity's Last Exam (HLE), a multi-modal benchmark at
  the frontier of human knowledge, designed to be the final closed-ended academic
  benchmark of its kind with broad subject coverage. HLE consists of 3,000
  questions across dozens of subjects, including mathematics, humanities, and the
  natural sciences. HLE is developed globally by subject-matter experts and
  consists of multiple-choice and short-answer questions suitable for automated
  grading. Each question has a known solution that is unambiguous and easily
  verifiable, but cannot be quickly answered via internet retrieval.
  State-of-the-art LLMs demonstrate low accuracy and calibration on HLE,
  highlighting a significant gap between current LLM capabilities and the expert
  human frontier on closed-ended academic questions. To inform research and
  policymaking upon a clear understanding of model capabilities, we publicly
  release HLE at https://lastexam.ai.
paper_authors._id: 679704b422b334a8370d3572
paper_authors.name: Long Phan
paper_authors._id: 679704b422b334a8370d3573
paper_authors.name: Alice Gatti
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-27T14:52:50.410Z
paper_authors._id: 64b6bd94047fd5611593694a
paper_authors.avatarUrl: /avatars/9dc8dd0229fa0467a1414b9f5ebbab0d.svg
paper_authors.fullname: Alice Gatti
paper_authors.type: user
paper_authors.user: alicegatti
paper_authors._id: 679704b422b334a8370d3574
paper_authors.name: Ziwen Han
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-27T14:52:59.398Z
paper_authors._id: 62ed40fe9b538d02b451c211
paper_authors.avatarUrl: /avatars/d5486b050fd63ade348158e64a3e5ff7.svg
paper_authors.fullname: Han
paper_authors.type: user
paper_authors.user: Ziwen
paper_authors._id: 679704b422b334a8370d3575
paper_authors.name: Nathaniel Li
paper_authors._id: 679704b422b334a8370d3576
paper_authors.name: Josephina Hu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-27T14:53:15.392Z
paper_authors._id: 66b13ca5c8900ac0175a7bc5
paper_authors.avatarUrl: /avatars/90e2c55145b1ee94fb6ef54426dfcdaa.svg
paper_authors.fullname: Josephina Hu
paper_authors.type: user
paper_authors.user: lovejosaay
paper_authors._id: 679704b422b334a8370d3577
paper_authors.name: Hugh Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-27T14:53:24.388Z
paper_authors._id: 65d6a5f94c28026a003581b4
paper_authors.avatarUrl: /avatars/c24c314eafbf6a885c586258ae87a47f.svg
paper_authors.fullname: Hugh Zhang
paper_authors.type: user
paper_authors.user: hugh-scale
paper_authors._id: 679704b422b334a8370d3578
paper_authors.name: Sean Shi
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-27T14:53:30.447Z
paper_authors._id: 66590a222d857c7a34771880
paper_authors.avatarUrl: /avatars/e6ff7d1b68feffe8953a62da380a60c8.svg
paper_authors.fullname: Sean Shi
paper_authors.type: user
paper_authors.user: seanshi-scale
paper_authors._id: 679704b422b334a8370d3579
paper_authors.name: Michael Choi
paper_authors._id: 679704b422b334a8370d357a
paper_authors.name: Anish Agrawal
paper_authors._id: 679704b422b334a8370d357b
paper_authors.name: Arnav Chopra
paper_authors._id: 679704b422b334a8370d357c
paper_authors.name: Adam Khoja
paper_authors._id: 679704b422b334a8370d357d
paper_authors.name: Ryan Kim
paper_authors._id: 679704b422b334a8370d357e
paper_authors.name: Jason Hausenloy
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-27T14:54:24.451Z
paper_authors._id: 6732f99ce6a45b6a0b322702
paper_authors.avatarUrl: /avatars/91a3266fe3f1a4c822739d9baba53c68.svg
paper_authors.fullname: Jason Hausenloy
paper_authors.type: user
paper_authors.user: jasonhausenloy
paper_authors._id: 679704b422b334a8370d357f
paper_authors.name: Oliver Zhang
paper_authors._id: 679704b422b334a8370d3580
paper_authors.name: Mantas Mazeika
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-27T14:54:56.624Z
paper_authors._id: 622bbdeb47e49335b2128a06
paper_authors.avatarUrl: /avatars/acaa6792da1a79a1ba91bbd148c85087.svg
paper_authors.fullname: Mantas Mazeika
paper_authors.type: user
paper_authors.user: mmazeika
paper_authors._id: 679704b422b334a8370d3581
paper_authors.name: Daron Anderson
paper_authors._id: 679704b422b334a8370d3582
paper_authors.name: Tung Nguyen
paper_authors._id: 679704b422b334a8370d3583
paper_authors.name: Mobeen Mahmood
paper_authors._id: 679704b422b334a8370d3584
paper_authors.name: Fiona Feng
paper_authors._id: 679704b422b334a8370d3585
paper_authors.name: Steven Y. Feng
paper_authors._id: 679704b422b334a8370d3586
paper_authors.name: Haoran Zhao
paper_authors._id: 679704b422b334a8370d3587
paper_authors.name: Michael Yu
paper_authors._id: 679704b422b334a8370d3588
paper_authors.name: Varun Gangal
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-27T14:55:42.966Z
paper_authors._id: 617ac050ff689804c90eca49
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1635434570481-noauth.jpeg
paper_authors.fullname: Varun Gangal
paper_authors.type: user
paper_authors.user: vgtomahawk
paper_authors._id: 679704b422b334a8370d3589
paper_authors.name: Chelsea Zou
paper_authors._id: 679704b422b334a8370d358a
paper_authors.name: Zihan Wang
paper_authors._id: 679704b422b334a8370d358b
paper_authors.name: Jessica P. Wang
paper_authors._id: 679704b422b334a8370d358c
paper_authors.name: Pawan Kumar
paper_authors._id: 679704b422b334a8370d358d
paper_authors.name: Oleksandr Pokutnyi
paper_authors._id: 679704b422b334a8370d358e
paper_authors.name: Robert Gerbicz
paper_authors._id: 679704b422b334a8370d358f
paper_authors.name: Serguei Popov
paper_authors._id: 679704b422b334a8370d3590
paper_authors.name: John-Clark Levin
paper_authors._id: 679704b422b334a8370d3591
paper_authors.name: Mstyslav Kazakov
paper_authors._id: 679704b422b334a8370d3592
paper_authors.name: Johannes Schmitt
paper_authors._id: 679704b422b334a8370d3593
paper_authors.name: Geoff Galgon
paper_authors._id: 679704b422b334a8370d3594
paper_authors.name: Alvaro Sanchez
paper_authors._id: 679704b422b334a8370d3595
paper_authors.name: Yongki Lee
paper_authors._id: 679704b422b334a8370d3596
paper_authors.name: Will Yeadon
paper_authors._id: 679704b422b334a8370d3597
paper_authors.name: Scott Sauers
paper_authors._id: 679704b422b334a8370d3598
paper_authors.name: Marc Roth
paper_authors._id: 679704b422b334a8370d3599
paper_authors.name: Chidozie Agu
paper_authors._id: 679704b422b334a8370d359a
paper_authors.name: Søren Riis
paper_authors._id: 679704b422b334a8370d359b
paper_authors.name: Fabian Giska
paper_authors._id: 679704b422b334a8370d359c
paper_authors.name: Saiteja Utpala
paper_authors._id: 679704b422b334a8370d359d
paper_authors.name: Zachary Giboney
paper_authors._id: 679704b422b334a8370d359e
paper_authors.name: Gashaw M. Goshu
paper_authors._id: 679704b422b334a8370d359f
paper_authors.name: Joan of Arc Xavier
paper_authors._id: 679704b422b334a8370d35a0
paper_authors.name: Sarah-Jane Crowson
paper_authors._id: 679704b422b334a8370d35a1
paper_authors.name: Mohinder Maheshbhai Naiya
paper_authors._id: 679704b422b334a8370d35a2
paper_authors.name: Noah Burns
paper_authors._id: 679704b422b334a8370d35a3
paper_authors.name: Lennart Finke
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-12T08:50:19.990Z
paper_authors._id: 66d823d3b61dd110220f80c3
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/66d823d3b61dd110220f80c3/ZiO99MNnWe3c-OnVWV-QB.png
paper_authors.fullname: Lennart Finke
paper_authors.type: user
paper_authors.user: lennart-finke
paper_authors._id: 679704b422b334a8370d35a4
paper_authors.name: Zerui Cheng
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-18T08:12:15.422Z
paper_authors._id: 67d8e5f041fc44bb16e5468c
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/jUCe0rf7Z-cF51OC5Iczn.png
paper_authors.fullname: Zerui Cheng
paper_authors.type: user
paper_authors.user: ZeruiCheng
paper_authors._id: 679704b422b334a8370d35a5
paper_authors.name: Hyunwoo Park
paper_authors._id: 679704b422b334a8370d35a6
paper_authors.name: Francesco Fournier-Facio
paper_authors._id: 679704b422b334a8370d35a7
paper_authors.name: John Wydallis
paper_authors._id: 679704b422b334a8370d35a8
paper_authors.name: Mark Nandor
paper_authors._id: 679704b422b334a8370d35a9
paper_authors.name: Ankit Singh
paper_authors._id: 679704b422b334a8370d35aa
paper_authors.name: Tim Gehrunger
paper_authors._id: 679704b422b334a8370d35ab
paper_authors.name: Jiaqi Cai
paper_authors._id: 679704b422b334a8370d35ac
paper_authors.name: Ben McCarty
paper_authors._id: 679704b422b334a8370d35ad
paper_authors.name: Darling Duclosel
paper_authors._id: 679704b422b334a8370d35ae
paper_authors.name: Jungbae Nam
paper_authors._id: 679704b422b334a8370d35af
paper_authors.name: Jennifer Zampese
paper_authors._id: 679704b422b334a8370d35b0
paper_authors.name: Ryan G. Hoerr
paper_authors._id: 679704b422b334a8370d35b1
paper_authors.name: Aras Bacho
paper_authors._id: 679704b422b334a8370d35b2
paper_authors.name: Gautier Abou Loume
paper_authors._id: 679704b422b334a8370d35b3
paper_authors.name: Abdallah Galal
paper_authors._id: 679704b422b334a8370d35b4
paper_authors.name: Hangrui Cao
paper_authors._id: 679704b422b334a8370d35b5
paper_authors.name: Alexis C Garretson
paper_authors._id: 679704b422b334a8370d35b6
paper_authors.name: Damien Sileo
paper_authors._id: 679704b422b334a8370d35b7
paper_authors.name: Qiuyu Ren
paper_authors._id: 679704b422b334a8370d35b8
paper_authors.name: Doru Cojoc
paper_authors._id: 679704b422b334a8370d35b9
paper_authors.name: Pavel Arkhipov
paper_authors._id: 679704b422b334a8370d35ba
paper_authors.name: Usman Qazi
paper_authors._id: 679704b422b334a8370d35bb
paper_authors.name: Lianghui Li
paper_authors._id: 679704b422b334a8370d35bc
paper_authors.name: Sumeet Motwani
paper_authors._id: 679704b422b334a8370d35bd
paper_authors.name: Christian Schroeder de Witt
paper_authors._id: 679704b422b334a8370d35be
paper_authors.name: Edwin Taylor
paper_authors._id: 679704b422b334a8370d35bf
paper_authors.name: Johannes Veith
paper_authors._id: 679704b422b334a8370d35c0
paper_authors.name: Eric Singer
paper_authors._id: 679704b422b334a8370d35c1
paper_authors.name: Taylor D. Hartman
paper_authors._id: 679704b422b334a8370d35c2
paper_authors.name: Paolo Rissone
paper_authors._id: 679704b422b334a8370d35c3
paper_authors.name: Jaehyeok Jin
paper_authors._id: 679704b422b334a8370d35c4
paper_authors.name: Jack Wei Lun Shi
paper_authors._id: 679704b422b334a8370d35c5
paper_authors.name: Chris G. Willcocks
paper_authors._id: 679704b422b334a8370d35c6
paper_authors.name: Joshua Robinson
paper_authors._id: 679704b422b334a8370d35c7
paper_authors.name: Aleksandar Mikov
paper_authors._id: 679704b422b334a8370d35c8
paper_authors.name: Ameya Prabhu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-27T12:55:03.605Z
paper_authors._id: 6464a0d41683d3c81f51924a
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6464a0d41683d3c81f51924a/s7yYVwfUB4WOhVFJS6A6T.jpeg
paper_authors.fullname: Ameya Prabhu
paper_authors.type: user
paper_authors.user: AmeyaPrabhu
paper_authors._id: 679704b422b334a8370d35c9
paper_authors.name: Longke Tang
paper_authors._id: 679704b422b334a8370d35ca
paper_authors.name: Xavier Alapont
paper_authors._id: 679704b422b334a8370d35cb
paper_authors.name: Justine Leon Uro
paper_authors._id: 679704b422b334a8370d35cc
paper_authors.name: Kevin Zhou
paper_authors._id: 679704b422b334a8370d35cd
paper_authors.name: Emily de Oliveira Santos
paper_authors._id: 679704b422b334a8370d35ce
paper_authors.name: Andrey Pupasov Maksimov
paper_authors._id: 679704b422b334a8370d35cf
paper_authors.name: Edward Vendrow
paper_authors._id: 679704b422b334a8370d35d0
paper_authors.name: Kengo Zenitani
paper_authors._id: 679704b422b334a8370d35d1
paper_authors.name: Julien Guillod
paper_authors._id: 679704b422b334a8370d35d2
paper_authors.name: Yuqi Li
paper_authors._id: 679704b422b334a8370d35d3
paper_authors.name: Joshua Vendrow
paper_authors._id: 679704b422b334a8370d35d4
paper_authors.name: Vladyslav Kuchkin
paper_authors._id: 679704b422b334a8370d35d5
paper_authors.name: Ng Ze-An
paper_authors._id: 679704b422b334a8370d35d6
paper_authors.name: Pierre Marion
paper_authors._id: 679704b422b334a8370d35d7
paper_authors.name: Denis Efremov
paper_authors._id: 679704b422b334a8370d35d8
paper_authors.name: Jayson Lynch
paper_authors._id: 679704b422b334a8370d35d9
paper_authors.name: Kaiqu Liang
paper_authors._id: 679704b422b334a8370d35da
paper_authors.name: Andrew Gritsevskiy
paper_authors._id: 679704b422b334a8370d35db
paper_authors.name: Dakotah Martinez
paper_authors._id: 679704b422b334a8370d35dc
paper_authors.name: Ben Pageler
paper_authors._id: 679704b422b334a8370d35dd
paper_authors.name: Nick Crispino
paper_authors._id: 679704b422b334a8370d35de
paper_authors.name: Dimitri Zvonkine
paper_authors._id: 679704b422b334a8370d35df
paper_authors.name: Natanael Wildner Fraga
paper_authors._id: 679704b422b334a8370d35e0
paper_authors.name: Saeed Soori
paper_authors._id: 679704b422b334a8370d35e1
paper_authors.name: Ori Press
paper_authors._id: 679704b422b334a8370d35e2
paper_authors.name: Henry Tang
paper_authors._id: 679704b422b334a8370d35e3
paper_authors.name: Julian Salazar
paper_authors._id: 679704b422b334a8370d35e4
paper_authors.name: Sean R. Green
paper_authors._id: 679704b422b334a8370d35e5
paper_authors.name: Lina Brüssel
paper_authors._id: 679704b422b334a8370d35e6
paper_authors.name: Moon Twayana
paper_authors._id: 679704b422b334a8370d35e7
paper_authors.name: Aymeric Dieuleveut
paper_authors._id: 679704b422b334a8370d35e8
paper_authors.name: T. Ryan Rogers
paper_authors._id: 679704b422b334a8370d35e9
paper_authors.name: Wenjin Zhang
paper_authors._id: 679704b422b334a8370d35ea
paper_authors.name: Bikun Li
paper_authors._id: 679704b422b334a8370d35eb
paper_authors.name: Jinzhou Yang
paper_authors._id: 679704b422b334a8370d35ec
paper_authors.name: Arun Rao
paper_authors._id: 679704b422b334a8370d35ed
paper_authors.name: Gabriel Loiseau
paper_authors._id: 679704b422b334a8370d35ee
paper_authors.name: Mikhail Kalinin
paper_authors._id: 679704b422b334a8370d35ef
paper_authors.name: Marco Lukas
paper_authors._id: 679704b422b334a8370d35f0
paper_authors.name: Ciprian Manolescu
paper_authors._id: 679704b422b334a8370d35f1
paper_authors.name: Subrata Mishra
paper_authors._id: 679704b422b334a8370d35f2
paper_authors.name: Ariel Ghislain Kemogne Kamdoum
paper_authors._id: 679704b422b334a8370d35f3
paper_authors.name: Tobias Kreiman
paper_authors._id: 679704b422b334a8370d35f4
paper_authors.name: Tad Hogg
paper_authors._id: 679704b422b334a8370d35f5
paper_authors.name: Alvin Jin
paper_authors._id: 679704b422b334a8370d35f6
paper_authors.name: Carlo Bosio
paper_authors._id: 679704b422b334a8370d35f7
paper_authors.name: Gongbo Sun
paper_authors._id: 679704b422b334a8370d35f8
paper_authors.name: Brian P Coppola
paper_authors._id: 679704b422b334a8370d35f9
paper_authors.name: Tim Tarver
paper_authors._id: 679704b422b334a8370d35fa
paper_authors.name: Haline Heidinger
paper_authors._id: 679704b422b334a8370d35fb
paper_authors.name: Rafael Sayous
paper_authors._id: 679704b422b334a8370d35fc
paper_authors.name: Stefan Ivanov
paper_authors._id: 679704b422b334a8370d35fd
paper_authors.name: Joseph M Cavanagh
paper_authors._id: 679704b422b334a8370d35fe
paper_authors.name: Jiawei Shen
paper_authors._id: 679704b422b334a8370d35ff
paper_authors.name: Joseph Marvin Imperial
paper_authors._id: 679704b422b334a8370d3600
paper_authors.name: Philippe Schwaller
paper_authors._id: 679704b422b334a8370d3601
paper_authors.name: Shaipranesh Senthilkuma
paper_authors._id: 679704b422b334a8370d3602
paper_authors.name: Andres M Bran
paper_authors._id: 679704b422b334a8370d3603
paper_authors.name: Ali Dehghan
paper_authors._id: 679704b422b334a8370d3604
paper_authors.name: Andres Algaba
paper_authors._id: 679704b422b334a8370d3605
paper_authors.name: Brecht Verbeken
paper_authors._id: 679704b422b334a8370d3606
paper_authors.name: David Noever
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-01-28T10:10:21.620Z
paper_authors._id: 63136a82e29fb2e86d5e5bdd
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/63136a82e29fb2e86d5e5bdd/pFZDuQtzfUStovbwwZGvn.png
paper_authors.fullname: David Noever
paper_authors.type: user
paper_authors.user: dnoever
paper_authors._id: 679704b422b334a8370d3607
paper_authors.name: Ragavendran P V
paper_authors._id: 679704b422b334a8370d3608
paper_authors.name: Lisa Schut
paper_authors._id: 679704b422b334a8370d3609
paper_authors.name: Ilia Sucholutsky
paper_authors._id: 679704b422b334a8370d360a
paper_authors.name: Evgenii Zheltonozhskii
paper_authors._id: 679704b422b334a8370d360b
paper_authors.name: Derek Lim
paper_authors._id: 679704b422b334a8370d360c
paper_authors.name: Richard Stanley
paper_authors._id: 679704b422b334a8370d360d
paper_authors.name: Shankar Sivarajan
paper_authors._id: 679704b422b334a8370d360e
paper_authors.name: Tong Yang
paper_authors._id: 679704b422b334a8370d360f
paper_authors.name: John Maar
paper_authors._id: 679704b422b334a8370d3610
paper_authors.name: Julian Wykowski
paper_authors._id: 679704b422b334a8370d3611
paper_authors.name: Martí Oller
paper_authors._id: 679704b422b334a8370d3612
paper_authors.name: Jennifer Sandlin
paper_authors._id: 679704b422b334a8370d3613
paper_authors.name: Anmol Sahu
paper_authors._id: 679704b422b334a8370d3614
paper_authors.name: Yuzheng Hu
paper_authors._id: 679704b422b334a8370d3615
paper_authors.name: Sara Fish
paper_authors._id: 679704b422b334a8370d3616
paper_authors.name: Nasser Heydari
paper_authors._id: 679704b422b334a8370d3617
paper_authors.name: Archimedes Apronti
paper_authors._id: 679704b422b334a8370d3618
paper_authors.name: Kaivalya Rawal
paper_authors._id: 679704b422b334a8370d3619
paper_authors.name: Tobias Garcia Vilchis
paper_authors._id: 679704b422b334a8370d361a
paper_authors.name: Yuexuan Zu
paper_authors._id: 679704b422b334a8370d361b
paper_authors.name: Martin Lackner
paper_authors._id: 679704b422b334a8370d361c
paper_authors.name: James Koppel
paper_authors._id: 679704b422b334a8370d361d
paper_authors.name: Jeremy Nguyen
paper_authors._id: 679704b422b334a8370d361e
paper_authors.name: Daniil S. Antonenko
paper_authors._id: 679704b422b334a8370d361f
paper_authors.name: Steffi Chern
paper_authors._id: 679704b422b334a8370d3620
paper_authors.name: Bingchen Zhao
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-01-28T10:10:25.495Z
paper_authors._id: 62dcd71075e9787ec5aa41ba
paper_authors.avatarUrl: /avatars/f37ce036b76180ed0fa004f9c8c09363.svg
paper_authors.fullname: Bingchen Zhao
paper_authors.type: user
paper_authors.user: tennant
paper_authors._id: 679704b422b334a8370d3621
paper_authors.name: Pierrot Arsene
paper_authors._id: 679704b422b334a8370d3622
paper_authors.name: Alan Goldfarb
paper_authors._id: 679704b422b334a8370d3623
paper_authors.name: Sergey Ivanov
paper_authors._id: 679704b422b334a8370d3624
paper_authors.name: Rafał Poświata
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-01-28T10:10:27.365Z
paper_authors._id: 63933543f8b4767ae646e8a1
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1670591762482-noauth.png
paper_authors.fullname: Rafał Poświata
paper_authors.type: user
paper_authors.user: rafalposwiata
paper_authors._id: 679704b422b334a8370d3625
paper_authors.name: Chenguang Wang
paper_authors._id: 679704b422b334a8370d3626
paper_authors.name: Daofeng Li
paper_authors._id: 679704b422b334a8370d3627
paper_authors.name: Donato Crisostomi
paper_authors._id: 679704b422b334a8370d3628
paper_authors.name: Andrea Achilleos
paper_authors._id: 679704b422b334a8370d3629
paper_authors.name: Benjamin Myklebust
paper_authors._id: 679704b422b334a8370d362a
paper_authors.name: Archan Sen
paper_authors._id: 679704b422b334a8370d362b
paper_authors.name: David Perrella
paper_authors._id: 679704b422b334a8370d362c
paper_authors.name: Nurdin Kaparov
paper_authors._id: 679704b422b334a8370d362d
paper_authors.name: Mark H Inlow
paper_authors._id: 679704b422b334a8370d362e
paper_authors.name: Allen Zang
paper_authors._id: 679704b422b334a8370d362f
paper_authors.name: Elliott Thornley
paper_authors._id: 679704b422b334a8370d3630
paper_authors.name: Daniil Orel
paper_authors._id: 679704b422b334a8370d3631
paper_authors.name: Vladislav Poritski
paper_authors._id: 679704b422b334a8370d3632
paper_authors.name: Shalev Ben-David
paper_authors._id: 679704b422b334a8370d3633
paper_authors.name: Zachary Berger
paper_authors._id: 679704b422b334a8370d3634
paper_authors.name: Parker Whitfill
paper_authors._id: 679704b422b334a8370d3635
paper_authors.name: Michael Foster
paper_authors._id: 679704b422b334a8370d3636
paper_authors.name: Daniel Munro
paper_authors._id: 679704b422b334a8370d3637
paper_authors.name: Linh Ho
paper_authors._id: 679704b422b334a8370d3638
paper_authors.name: Dan Bar Hava
paper_authors._id: 679704b422b334a8370d3639
paper_authors.name: Aleksey Kuchkin
paper_authors._id: 679704b422b334a8370d363a
paper_authors.name: Robert Lauff
paper_authors._id: 679704b422b334a8370d363b
paper_authors.name: David Holmes
paper_authors._id: 679704b422b334a8370d363c
paper_authors.name: Frank Sommerhage
paper_authors._id: 679704b422b334a8370d363d
paper_authors.name: Keith Schneider
paper_authors._id: 679704b422b334a8370d363e
paper_authors.name: Zakayo Kazibwe
paper_authors._id: 679704b422b334a8370d363f
paper_authors.name: Nate Stambaugh
paper_authors._id: 679704b422b334a8370d3640
paper_authors.name: Mukhwinder Singh
paper_authors._id: 679704b422b334a8370d3641
paper_authors.name: Ilias Magoulas
paper_authors._id: 679704b422b334a8370d3642
paper_authors.name: Don Clarke
paper_authors._id: 679704b422b334a8370d3643
paper_authors.name: Dae Hyun Kim
paper_authors._id: 679704b422b334a8370d3644
paper_authors.name: Felipe Meneguitti Dias
paper_authors._id: 679704b422b334a8370d3645
paper_authors.name: Veit Elser
paper_authors._id: 679704b422b334a8370d3646
paper_authors.name: Kanu Priya Agarwal
paper_authors._id: 679704b422b334a8370d3647
paper_authors.name: Victor Efren Guadarrama Vilchis
paper_authors._id: 679704b422b334a8370d3648
paper_authors.name: Immo Klose
paper_authors._id: 679704b422b334a8370d3649
paper_authors.name: Christoph Demian
paper_authors._id: 679704b422b334a8370d364a
paper_authors.name: Ujjwala Anantheswaran
paper_authors._id: 679704b422b334a8370d364b
paper_authors.name: Adam Zweiger
paper_authors._id: 679704b422b334a8370d364c
paper_authors.name: Guglielmo Albani
paper_authors._id: 679704b422b334a8370d364d
paper_authors.name: Jeffery Li
paper_authors._id: 679704b422b334a8370d364e
paper_authors.name: Nicolas Daans
paper_authors._id: 679704b422b334a8370d364f
paper_authors.name: Maksim Radionov
paper_authors._id: 679704b422b334a8370d3650
paper_authors.name: Václav Rozhoň
paper_authors._id: 679704b422b334a8370d3651
paper_authors.name: Ziqiao Ma
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-27T22:09:25.436Z
paper_authors._id: 630cfc45b66f088d547b2768
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/630cfc45b66f088d547b2768/ge2BZLDYeTlpnlIlraHNw.png
paper_authors.fullname: Martin Ziqiao Ma
paper_authors.type: user
paper_authors.user: marstin
paper_authors._id: 679704b422b334a8370d3652
paper_authors.name: Christian Stump
paper_authors._id: 679704b422b334a8370d3653
paper_authors.name: Mohammed Berkani
paper_authors._id: 679704b422b334a8370d3654
paper_authors.name: Jacob Platnick
paper_authors._id: 679704b422b334a8370d3655
paper_authors.name: Volodymyr Nevirkovets
paper_authors._id: 679704b422b334a8370d3656
paper_authors.name: Luke Basler
paper_authors._id: 679704b422b334a8370d3657
paper_authors.name: Marco Piccardo
paper_authors._id: 679704b422b334a8370d3658
paper_authors.name: Ferenc Jeanplong
paper_authors._id: 679704b422b334a8370d3659
paper_authors.name: Niv Cohen
paper_authors._id: 679704b422b334a8370d365a
paper_authors.name: Josef Tkadlec
paper_authors._id: 679704b422b334a8370d365b
paper_authors.name: Paul Rosu
paper_authors._id: 679704b422b334a8370d365c
paper_authors.name: Piotr Padlewski
paper_authors._id: 679704b422b334a8370d365d
paper_authors.name: Stanislaw Barzowski
paper_authors._id: 679704b422b334a8370d365e
paper_authors.name: Kyle Montgomery
paper_authors._id: 679704b422b334a8370d365f
paper_authors.name: Aline Menezes
paper_authors._id: 679704b422b334a8370d3660
paper_authors.name: Arkil Patel
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-24T09:26:54.744Z
paper_authors._id: 631a523c04f8ed65eff16fb4
paper_authors.avatarUrl: /avatars/2b284403c88f140d7bef283f729f7a3e.svg
paper_authors.fullname: Arkil Patel
paper_authors.type: user
paper_authors.user: arkilpatel
paper_authors._id: 679704b422b334a8370d3661
paper_authors.name: Zixuan Wang
paper_authors._id: 679704b422b334a8370d3662
paper_authors.name: Jamie Tucker-Foltz
paper_authors._id: 679704b422b334a8370d3663
paper_authors.name: Jack Stade
paper_authors._id: 679704b422b334a8370d3664
paper_authors.name: Tom Goertzen
paper_authors._id: 679704b422b334a8370d3665
paper_authors.name: Fereshteh Kazemi
paper_authors._id: 679704b422b334a8370d3666
paper_authors.name: Jeremiah Milbauer
paper_authors._id: 679704b422b334a8370d3667
paper_authors.name: John Arnold Ambay
paper_authors._id: 679704b422b334a8370d3668
paper_authors.name: Abhishek Shukla
paper_authors._id: 679704b422b334a8370d3669
paper_authors.name: Yan Carlos Leyva Labrador
paper_authors._id: 679704b422b334a8370d366a
paper_authors.name: Alan Givré
paper_authors._id: 679704b422b334a8370d366b
paper_authors.name: Hew Wolff
paper_authors._id: 679704b422b334a8370d366c
paper_authors.name: Vivien Rossbach
paper_authors._id: 679704b422b334a8370d366d
paper_authors.name: Muhammad Fayez Aziz
paper_authors._id: 679704b422b334a8370d366e
paper_authors.name: Younesse Kaddar
paper_authors._id: 679704b422b334a8370d366f
paper_authors.name: Yanxu Chen
paper_authors._id: 679704b422b334a8370d3670
paper_authors.name: Robin Zhang
paper_authors._id: 679704b422b334a8370d3671
paper_authors.name: Jiayi Pan
paper_authors._id: 679704b422b334a8370d3672
paper_authors.name: Antonio Terpin
paper_authors._id: 679704b422b334a8370d3673
paper_authors.name: Niklas Muennighoff
paper_authors._id: 679704b422b334a8370d3674
paper_authors.name: Hailey Schoelkopf
paper_authors._id: 679704b422b334a8370d3675
paper_authors.name: Eric Zheng
paper_authors._id: 679704b422b334a8370d3676
paper_authors.name: Avishy Carmi
paper_authors._id: 679704b422b334a8370d3677
paper_authors.name: Adam Jones
paper_authors._id: 679704b422b334a8370d3678
paper_authors.name: Jainam Shah
paper_authors._id: 679704b422b334a8370d3679
paper_authors.name: Ethan D. L. Brown
paper_authors._id: 679704b422b334a8370d367a
paper_authors.name: Kelin Zhu
paper_authors._id: 679704b422b334a8370d367b
paper_authors.name: Max Bartolo
paper_authors._id: 679704b422b334a8370d367c
paper_authors.name: Richard Wheeler
paper_authors._id: 679704b422b334a8370d367d
paper_authors.name: Andrew Ho
paper_authors._id: 679704b422b334a8370d367e
paper_authors.name: Shaul Barkan
paper_authors._id: 679704b422b334a8370d367f
paper_authors.name: Jiaqi Wang
paper_authors._id: 679704b422b334a8370d3680
paper_authors.name: Martin Stehberger
paper_authors._id: 679704b422b334a8370d3681
paper_authors.name: Egor Kretov
paper_authors._id: 679704b422b334a8370d3682
paper_authors.name: Kaustubh Sridhar
paper_authors._id: 679704b422b334a8370d3683
paper_authors.name: Zienab EL-Wasif
paper_authors._id: 679704b422b334a8370d3684
paper_authors.name: Anji Zhang
paper_authors._id: 679704b422b334a8370d3685
paper_authors.name: Daniel Pyda
paper_authors._id: 679704b422b334a8370d3686
paper_authors.name: Joanna Tam
paper_authors._id: 679704b422b334a8370d3687
paper_authors.name: David M. Cunningham
paper_authors._id: 679704b422b334a8370d3688
paper_authors.name: Vladimir Goryachev
paper_authors._id: 679704b422b334a8370d3689
paper_authors.name: Demosthenes Patramanis
paper_authors._id: 679704b422b334a8370d368a
paper_authors.name: Michael Krause
paper_authors._id: 679704b422b334a8370d368b
paper_authors.name: Andrew Redenti
paper_authors._id: 679704b422b334a8370d368c
paper_authors.name: Daniel Bugas
paper_authors._id: 679704b422b334a8370d368d
paper_authors.name: David Aldous
paper_authors._id: 679704b422b334a8370d368e
paper_authors.name: Jesyin Lai
paper_authors._id: 679704b422b334a8370d368f
paper_authors.name: Shannon Coleman
paper_authors._id: 679704b422b334a8370d3690
paper_authors.name: Mohsen Bahaloo
paper_authors._id: 679704b422b334a8370d3691
paper_authors.name: Jiangnan Xu
paper_authors._id: 679704b422b334a8370d3692
paper_authors.name: Sangwon Lee
paper_authors._id: 679704b422b334a8370d3693
paper_authors.name: Sandy Zhao
paper_authors._id: 679704b422b334a8370d3694
paper_authors.name: Ning Tang
paper_authors._id: 679704b422b334a8370d3695
paper_authors.name: Michael K. Cohen
paper_authors._id: 679704b422b334a8370d3696
paper_authors.name: Micah Carroll
paper_authors._id: 679704b422b334a8370d3697
paper_authors.name: Orr Paradise
paper_authors._id: 679704b422b334a8370d3698
paper_authors.name: Jan Hendrik Kirchner
paper_authors._id: 679704b422b334a8370d3699
paper_authors.name: Stefan Steinerberger
paper_authors._id: 679704b422b334a8370d369a
paper_authors.name: Maksym Ovchynnikov
paper_authors._id: 679704b422b334a8370d369b
paper_authors.name: Jason O. Matos
paper_authors._id: 679704b422b334a8370d369c
paper_authors.name: Adithya Shenoy
paper_authors._id: 679704b422b334a8370d369d
paper_authors.name: Benedito Alves de Oliveira Junior
paper_authors._id: 679704b422b334a8370d369e
paper_authors.name: Michael Wang
paper_authors._id: 679704b422b334a8370d369f
paper_authors.name: Yuzhou Nie
paper_authors._id: 679704b422b334a8370d36a0
paper_authors.name: Paolo Giordano
paper_authors._id: 679704b422b334a8370d36a1
paper_authors.name: Philipp Petersen
paper_authors._id: 679704b422b334a8370d36a2
paper_authors.name: Anna Sztyber-Betley
paper_authors._id: 679704b422b334a8370d36a3
paper_authors.name: Priti Shukla
paper_authors._id: 679704b422b334a8370d36a4
paper_authors.name: Jonathan Crozier
paper_authors._id: 679704b422b334a8370d36a5
paper_authors.name: Antonella Pinto
paper_authors._id: 679704b422b334a8370d36a6
paper_authors.name: Shreyas Verma
paper_authors._id: 679704b422b334a8370d36a7
paper_authors.name: Prashant Joshi
paper_authors._id: 679704b422b334a8370d36a8
paper_authors.name: Zheng-Xin Yong
paper_authors._id: 679704b422b334a8370d36a9
paper_authors.name: Allison Tee
paper_authors._id: 679704b422b334a8370d36aa
paper_authors.name: Jérémy Andréoletti
paper_authors._id: 679704b422b334a8370d36ab
paper_authors.name: Orion Weller
paper_authors._id: 679704b422b334a8370d36ac
paper_authors.name: Raghav Singhal
paper_authors._id: 679704b422b334a8370d36ad
paper_authors.name: Gang Zhang
paper_authors._id: 679704b422b334a8370d36ae
paper_authors.name: Alexander Ivanov
paper_authors._id: 679704b422b334a8370d36af
paper_authors.name: Seri Khoury
paper_authors._id: 679704b422b334a8370d36b0
paper_authors.name: Hamid Mostaghimi
paper_authors._id: 679704b422b334a8370d36b1
paper_authors.name: Kunvar Thaman
paper_authors._id: 679704b422b334a8370d36b2
paper_authors.name: Qijia Chen
paper_authors._id: 679704b422b334a8370d36b3
paper_authors.name: Tran Quoc Khánh
paper_authors._id: 679704b422b334a8370d36b4
paper_authors.name: Jacob Loader
paper_authors._id: 679704b422b334a8370d36b5
paper_authors.name: Stefano Cavalleri
paper_authors._id: 679704b422b334a8370d36b6
paper_authors.name: Hannah Szlyk
paper_authors._id: 679704b422b334a8370d36b7
paper_authors.name: Zachary Brown
paper_authors._id: 679704b422b334a8370d36b8
paper_authors.name: Jonathan Roberts
paper_authors._id: 679704b422b334a8370d36b9
paper_authors.name: William Alley
paper_authors._id: 679704b422b334a8370d36ba
paper_authors.name: Kunyang Sun
paper_authors._id: 679704b422b334a8370d36bb
paper_authors.name: Ryan Stendall
paper_authors._id: 679704b422b334a8370d36bc
paper_authors.name: Max Lamparth
paper_authors._id: 679704b422b334a8370d36bd
paper_authors.name: Anka Reuel
paper_authors._id: 679704b422b334a8370d36be
paper_authors.name: Ting Wang
paper_authors._id: 679704b422b334a8370d36bf
paper_authors.name: Hanmeng Xu
paper_authors._id: 679704b422b334a8370d36c0
paper_authors.name: Sreenivas Goud Raparthi
paper_authors._id: 679704b422b334a8370d36c1
paper_authors.name: Pablo Hernández-Cámara
paper_authors._id: 679704b422b334a8370d36c2
paper_authors.name: Freddie Martin
paper_authors._id: 679704b422b334a8370d36c3
paper_authors.name: Dmitry Malishev
paper_authors._id: 679704b422b334a8370d36c4
paper_authors.name: Thomas Preu
paper_authors._id: 679704b422b334a8370d36c5
paper_authors.name: Tomek Korbak
paper_authors._id: 679704b422b334a8370d36c6
paper_authors.name: Marcus Abramovitch
paper_authors._id: 679704b422b334a8370d36c7
paper_authors.name: Dominic Williamson
paper_authors._id: 679704b422b334a8370d36c8
paper_authors.name: Ziye Chen
paper_authors._id: 679704b422b334a8370d36c9
paper_authors.name: Biró Bálint
paper_authors._id: 679704b422b334a8370d36ca
paper_authors.name: M Saiful Bari
paper_authors._id: 679704b422b334a8370d36cb
paper_authors.name: Peyman Kassani
paper_authors._id: 679704b422b334a8370d36cc
paper_authors.name: Zihao Wang
paper_authors._id: 679704b422b334a8370d36cd
paper_authors.name: Behzad Ansarinejad
paper_authors._id: 679704b422b334a8370d36ce
paper_authors.name: Laxman Prasad Goswami
paper_authors._id: 679704b422b334a8370d36cf
paper_authors.name: Yewen Sun
paper_authors._id: 679704b422b334a8370d36d0
paper_authors.name: Hossam Elgnainy
paper_authors._id: 679704b422b334a8370d36d1
paper_authors.name: Daniel Tordera
paper_authors._id: 679704b422b334a8370d36d2
paper_authors.name: George Balabanian
paper_authors._id: 679704b422b334a8370d36d3
paper_authors.name: Earth Anderson
paper_authors._id: 679704b422b334a8370d36d4
paper_authors.name: Lynna Kvistad
paper_authors._id: 679704b422b334a8370d36d5
paper_authors.name: Alejandro José Moyano
paper_authors._id: 679704b422b334a8370d36d6
paper_authors.name: Rajat Maheshwari
paper_authors._id: 679704b422b334a8370d36d7
paper_authors.name: Ahmad Sakor
paper_authors._id: 679704b422b334a8370d36d8
paper_authors.name: Murat Eron
paper_authors._id: 679704b422b334a8370d36d9
paper_authors.name: Isaac C. McAlister
paper_authors._id: 679704b422b334a8370d36da
paper_authors.name: Javier Gimenez
paper_authors._id: 679704b422b334a8370d36db
paper_authors.name: Innocent Enyekwe
paper_authors._id: 679704b422b334a8370d36dc
paper_authors.name: Andrew Favre D. O.
paper_authors._id: 679704b422b334a8370d36dd
paper_authors.name: Shailesh Shah
paper_authors._id: 679704b422b334a8370d36de
paper_authors.name: Xiaoxiang Zhou
paper_authors._id: 679704b422b334a8370d36df
paper_authors.name: Firuz Kamalov
paper_authors._id: 679704b422b334a8370d36e0
paper_authors.name: Ronald Clark
paper_authors._id: 679704b422b334a8370d36e1
paper_authors.name: Sherwin Abdoli
paper_authors._id: 679704b422b334a8370d36e2
paper_authors.name: Tim Santens
paper_authors._id: 679704b422b334a8370d36e3
paper_authors.name: Khalida Meer
paper_authors._id: 679704b422b334a8370d36e4
paper_authors.name: Harrison K Wang
paper_authors._id: 679704b422b334a8370d36e5
paper_authors.name: Kalyan Ramakrishnan
paper_authors._id: 679704b422b334a8370d36e6
paper_authors.name: Evan Chen
paper_authors._id: 679704b422b334a8370d36e7
paper_authors.name: Alessandro Tomasiello
paper_authors._id: 679704b422b334a8370d36e8
paper_authors.name: G. Bruno De Luca
paper_authors._id: 679704b422b334a8370d36e9
paper_authors.name: Shi-Zhuo Looi
paper_authors._id: 679704b422b334a8370d36ea
paper_authors.name: Vinh-Kha Le
paper_authors._id: 679704b422b334a8370d36eb
paper_authors.name: Noam Kolt
paper_authors._id: 679704b422b334a8370d36ec
paper_authors.name: Niels Mündler
paper_authors._id: 679704b422b334a8370d36ed
paper_authors.name: Avi Semler
paper_authors._id: 679704b422b334a8370d36ee
paper_authors.name: Emma Rodman
paper_authors._id: 679704b422b334a8370d36ef
paper_authors.name: Jacob Drori
paper_authors._id: 679704b422b334a8370d36f0
paper_authors.name: Carl J Fossum
paper_authors._id: 679704b422b334a8370d36f1
paper_authors.name: Milind Jagota
paper_authors._id: 679704b422b334a8370d36f2
paper_authors.name: Ronak Pradeep
paper_authors._id: 679704b422b334a8370d36f3
paper_authors.name: Honglu Fan
paper_authors._id: 679704b422b334a8370d36f4
paper_authors.name: Tej Shah
paper_authors._id: 679704b422b334a8370d36f5
paper_authors.name: Jonathan Eicher
paper_authors._id: 679704b422b334a8370d36f6
paper_authors.name: Michael Chen
paper_authors._id: 679704b422b334a8370d36f7
paper_authors.name: Kushal Thaman
paper_authors._id: 679704b422b334a8370d36f8
paper_authors.name: William Merrill
paper_authors._id: 679704b422b334a8370d36f9
paper_authors.name: Carter Harris
paper_authors._id: 679704b422b334a8370d36fa
paper_authors.name: Jason Gross
paper_authors._id: 679704b422b334a8370d36fb
paper_authors.name: Ilya Gusev
paper_authors._id: 679704b422b334a8370d36fc
paper_authors.name: Asankhaya Sharma
paper_authors._id: 679704b422b334a8370d36fd
paper_authors.name: Shashank Agnihotri
paper_authors._id: 679704b422b334a8370d36fe
paper_authors.name: Pavel Zhelnov
paper_authors._id: 679704b422b334a8370d36ff
paper_authors.name: Siranut Usawasutsakorn
paper_authors._id: 679704b422b334a8370d3700
paper_authors.name: Mohammadreza Mofayezi
paper_authors._id: 679704b422b334a8370d3701
paper_authors.name: Sergei Bogdanov
paper_authors._id: 679704b422b334a8370d3702
paper_authors.name: Alexander Piperski
paper_authors._id: 679704b422b334a8370d3703
paper_authors.name: Marc Carauleanu
paper_authors._id: 679704b422b334a8370d3704
paper_authors.name: David K. Zhang
paper_authors._id: 679704b422b334a8370d3705
paper_authors.name: Dylan Ler
paper_authors._id: 679704b422b334a8370d3706
paper_authors.name: Roman Leventov
paper_authors._id: 679704b422b334a8370d3707
paper_authors.name: Ignat Soroko
paper_authors._id: 679704b422b334a8370d3708
paper_authors.name: Thorben Jansen
paper_authors._id: 679704b422b334a8370d3709
paper_authors.name: Pascal Lauer
paper_authors._id: 679704b422b334a8370d370a
paper_authors.name: Joshua Duersch
paper_authors._id: 679704b422b334a8370d370b
paper_authors.name: Vage Taamazyan
paper_authors._id: 679704b422b334a8370d370c
paper_authors.name: Wiktor Morak
paper_authors._id: 679704b422b334a8370d370d
paper_authors.name: Wenjie Ma
paper_authors._id: 679704b422b334a8370d370e
paper_authors.name: William Held
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-26T08:42:57.161Z
paper_authors._id: 632116accafe12f481a473cb
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1666207676653-632116accafe12f481a473cb.jpeg
paper_authors.fullname: Will Held
paper_authors.type: user
paper_authors.user: WillHeld
paper_authors._id: 679704b422b334a8370d370f
paper_authors.name: Tran Đuc Huy
paper_authors._id: 679704b422b334a8370d3710
paper_authors.name: Ruicheng Xian
paper_authors._id: 679704b422b334a8370d3711
paper_authors.name: Armel Randy Zebaze
paper_authors._id: 679704b422b334a8370d3712
paper_authors.name: Mohanad Mohamed
paper_authors._id: 679704b422b334a8370d3713
paper_authors.name: Julian Noah Leser
paper_authors._id: 679704b422b334a8370d3714
paper_authors.name: Michelle X Yuan
paper_authors._id: 679704b422b334a8370d3715
paper_authors.name: Laila Yacar
paper_authors._id: 679704b422b334a8370d3716
paper_authors.name: Johannes Lengler
paper_authors._id: 679704b422b334a8370d3717
paper_authors.name: Hossein Shahrtash
paper_authors._id: 679704b422b334a8370d3718
paper_authors.name: Edson Oliveira
paper_authors._id: 679704b422b334a8370d3719
paper_authors.name: Joseph W. Jackson
paper_authors._id: 679704b422b334a8370d371a
paper_authors.name: Daniel Espinosa Gonzalez
paper_authors._id: 679704b422b334a8370d371b
paper_authors.name: Andy Zou
paper_authors._id: 679704b422b334a8370d371c
paper_authors.name: Muthu Chidambaram
paper_authors._id: 679704b422b334a8370d371d
paper_authors.name: Timothy Manik
paper_authors._id: 679704b422b334a8370d371e
paper_authors.name: Hector Haffenden
paper_authors._id: 679704b422b334a8370d371f
paper_authors.name: Dashiell Stander
paper_authors._id: 679704b422b334a8370d3720
paper_authors.name: Ali Dasouqi
paper_authors._id: 679704b422b334a8370d3721
paper_authors.name: Alexander Shen
paper_authors._id: 679704b422b334a8370d3722
paper_authors.name: Emilien Duc
paper_authors._id: 679704b422b334a8370d3723
paper_authors.name: Bita Golshani
paper_authors._id: 679704b422b334a8370d3724
paper_authors.name: David Stap
paper_authors._id: 679704b422b334a8370d3725
paper_authors.name: Mikalai Uzhou
paper_authors._id: 679704b422b334a8370d3726
paper_authors.name: Alina Borisovna Zhidkovskaya
paper_authors._id: 679704b422b334a8370d3727
paper_authors.name: Lukas Lewark
paper_authors._id: 679704b422b334a8370d3728
paper_authors.name: Mátyás Vincze
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-03T08:16:10.413Z
paper_authors._id: 6454203c548f22be59825ee3
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/kfKhnt59IiCvVoPS8JP3-.jpeg
paper_authors.fullname: Mátyás Vincze
paper_authors.type: user
paper_authors.user: vinczematyas
paper_authors._id: 679704b422b334a8370d3729
paper_authors.name: Dustin Wehr
paper_authors._id: 679704b422b334a8370d372a
paper_authors.name: Colin Tang
paper_authors._id: 679704b422b334a8370d372b
paper_authors.name: Zaki Hossain
paper_authors._id: 679704b422b334a8370d372c
paper_authors.name: Shaun Phillips
paper_authors._id: 679704b422b334a8370d372d
paper_authors.name: Jiang Muzhen
paper_authors._id: 679704b422b334a8370d372e
paper_authors.name: Fredrik Ekström
paper_authors._id: 679704b422b334a8370d372f
paper_authors.name: Angela Hammon
paper_authors._id: 679704b422b334a8370d3730
paper_authors.name: Oam Patel
paper_authors._id: 679704b422b334a8370d3731
paper_authors.name: Nicolas Remy
paper_authors._id: 679704b422b334a8370d3732
paper_authors.name: Faraz Farhidi
paper_authors._id: 679704b422b334a8370d3733
paper_authors.name: George Medley
paper_authors._id: 679704b422b334a8370d3734
paper_authors.name: Forough Mohammadzadeh
paper_authors._id: 679704b422b334a8370d3735
paper_authors.name: Madellene Peñaflor
paper_authors._id: 679704b422b334a8370d3736
paper_authors.name: Haile Kassahun
paper_authors._id: 679704b422b334a8370d3737
paper_authors.name: Alena Friedrich
paper_authors._id: 679704b422b334a8370d3738
paper_authors.name: Claire Sparrow
paper_authors._id: 679704b422b334a8370d3739
paper_authors.name: Taom Sakal
paper_authors._id: 679704b422b334a8370d373a
paper_authors.name: Omkar Dhamane
paper_authors._id: 679704b422b334a8370d373b
paper_authors.name: Ali Khajegili Mirabadi
paper_authors._id: 679704b422b334a8370d373c
paper_authors.name: Eric Hallman
paper_authors._id: 679704b422b334a8370d373d
paper_authors.name: Mike Battaglia
paper_authors._id: 679704b422b334a8370d373e
paper_authors.name: Mohammad Maghsoudimehrabani
paper_authors._id: 679704b422b334a8370d373f
paper_authors.name: Hieu Hoang
paper_authors._id: 679704b422b334a8370d3740
paper_authors.name: Alon Amit
paper_authors._id: 679704b422b334a8370d3741
paper_authors.name: Dave Hulbert
paper_authors._id: 679704b422b334a8370d3742
paper_authors.name: Roberto Pereira
paper_authors._id: 679704b422b334a8370d3743
paper_authors.name: Simon Weber
paper_authors._id: 679704b422b334a8370d3744
paper_authors.name: Stephen Mensah
paper_authors._id: 679704b422b334a8370d3745
paper_authors.name: Nathan Andre
paper_authors._id: 679704b422b334a8370d3746
paper_authors.name: Anton Peristyy
paper_authors._id: 679704b422b334a8370d3747
paper_authors.name: Chris Harjadi
paper_authors._id: 679704b422b334a8370d3748
paper_authors.name: Himanshu Gupta
paper_authors._id: 679704b422b334a8370d3749
paper_authors.name: Stephen Malina
paper_authors._id: 679704b422b334a8370d374a
paper_authors.name: Samuel Albanie
paper_authors._id: 679704b422b334a8370d374b
paper_authors.name: Will Cai
paper_authors._id: 679704b422b334a8370d374c
paper_authors.name: Mustafa Mehkary
paper_authors._id: 679704b422b334a8370d374d
paper_authors.name: Frank Reidegeld
paper_authors._id: 679704b422b334a8370d374e
paper_authors.name: Anna-Katharina Dick
paper_authors._id: 679704b422b334a8370d374f
paper_authors.name: Cary Friday
paper_authors._id: 679704b422b334a8370d3750
paper_authors.name: Jasdeep Sidhu
paper_authors._id: 679704b422b334a8370d3751
paper_authors.name: Wanyoung Kim
paper_authors._id: 679704b422b334a8370d3752
paper_authors.name: Mariana Costa
paper_authors._id: 679704b422b334a8370d3753
paper_authors.name: Hubeyb Gurdogan
paper_authors._id: 679704b422b334a8370d3754
paper_authors.name: Brian Weber
paper_authors._id: 679704b422b334a8370d3755
paper_authors.name: Harsh Kumar
paper_authors._id: 679704b422b334a8370d3756
paper_authors.name: Tong Jiang
paper_authors._id: 679704b422b334a8370d3757
paper_authors.name: Arunim Agarwal
paper_authors._id: 679704b422b334a8370d3758
paper_authors.name: Chiara Ceconello
paper_authors._id: 679704b422b334a8370d3759
paper_authors.name: Warren S. Vaz
paper_authors._id: 679704b422b334a8370d375a
paper_authors.name: Chao Zhuang
paper_authors._id: 679704b422b334a8370d375b
paper_authors.name: Haon Park
paper_authors._id: 679704b422b334a8370d375c
paper_authors.name: Andrew R. Tawfeek
paper_authors._id: 679704b422b334a8370d375d
paper_authors.name: Daattavya Aggarwal
paper_authors._id: 679704b422b334a8370d375e
paper_authors.name: Michael Kirchhof
paper_authors._id: 679704b422b334a8370d375f
paper_authors.name: Linjie Dai
paper_authors._id: 679704b422b334a8370d3760
paper_authors.name: Evan Kim
paper_authors._id: 679704b422b334a8370d3761
paper_authors.name: Johan Ferret
paper_authors._id: 679704b422b334a8370d3762
paper_authors.name: Yuzhou Wang
paper_authors._id: 679704b422b334a8370d3763
paper_authors.name: Minghao Yan
paper_authors._id: 679704b422b334a8370d3764
paper_authors.name: Krzysztof Burdzy
paper_authors._id: 679704b422b334a8370d3765
paper_authors.name: Lixin Zhang
paper_authors._id: 679704b422b334a8370d3766
paper_authors.name: Antonio Franca
paper_authors._id: 679704b422b334a8370d3767
paper_authors.name: Diana T. Pham
paper_authors._id: 679704b422b334a8370d3768
paper_authors.name: Kang Yong Loh
paper_authors._id: 679704b422b334a8370d3769
paper_authors.name: Joshua Robinson
paper_authors._id: 679704b422b334a8370d376a
paper_authors.name: Shreen Gul
paper_authors._id: 679704b422b334a8370d376b
paper_authors.name: Gunjan Chhablani
paper_authors._id: 679704b422b334a8370d376c
paper_authors.name: Zhehang Du
paper_authors._id: 679704b422b334a8370d376d
paper_authors.name: Adrian Cosma
paper_authors._id: 679704b422b334a8370d376e
paper_authors.name: Colin White
paper_authors._id: 679704b422b334a8370d376f
paper_authors.name: Robin Riblet
paper_authors._id: 679704b422b334a8370d3770
paper_authors.name: Prajvi Saxena
paper_authors._id: 679704b422b334a8370d3771
paper_authors.name: Jacob Votava
paper_authors._id: 679704b422b334a8370d3772
paper_authors.name: Vladimir Vinnikov
paper_authors._id: 679704b422b334a8370d3773
paper_authors.name: Ethan Delaney
paper_authors._id: 679704b422b334a8370d3774
paper_authors.name: Shiv Halasyamani
paper_authors._id: 679704b422b334a8370d3775
paper_authors.name: Syed M. Shahid
paper_authors._id: 679704b422b334a8370d3776
paper_authors.name: Jean-Christophe Mourrat
paper_authors._id: 679704b422b334a8370d3777
paper_authors.name: Lavr Vetoshkin
paper_authors._id: 679704b422b334a8370d3778
paper_authors.name: Renas Bacho
paper_authors._id: 679704b422b334a8370d3779
paper_authors.name: Vincent Ginis
paper_authors._id: 679704b422b334a8370d377a
paper_authors.name: Aleksandr Maksapetyan
paper_authors._id: 679704b422b334a8370d377b
paper_authors.name: Florencia de la Rosa
paper_authors._id: 679704b422b334a8370d377c
paper_authors.name: Xiuyu Li
paper_authors._id: 679704b422b334a8370d377d
paper_authors.name: Guillaume Malod
paper_authors._id: 679704b422b334a8370d377e
paper_authors.name: Leon Lang
paper_authors._id: 679704b422b334a8370d377f
paper_authors.name: Julien Laurendeau
paper_authors._id: 679704b422b334a8370d3780
paper_authors.name: Fatimah Adesanya
paper_authors._id: 679704b422b334a8370d3781
paper_authors.name: Julien Portier
paper_authors._id: 679704b422b334a8370d3782
paper_authors.name: Lawrence Hollom
paper_authors._id: 679704b422b334a8370d3783
paper_authors.name: Victor Souza
paper_authors._id: 679704b422b334a8370d3784
paper_authors.name: Yuchen Anna Zhou
paper_authors._id: 679704b422b334a8370d3785
paper_authors.name: Yiğit Yalın
paper_authors._id: 679704b422b334a8370d3786
paper_authors.name: Gbenga Daniel Obikoya
paper_authors._id: 679704b422b334a8370d3787
paper_authors.name: Luca Arnaboldi
paper_authors._id: 679704b422b334a8370d3788
paper_authors.name: Rai
paper_authors._id: 679704b422b334a8370d3789
paper_authors.name: Filippo Bigi
paper_authors._id: 679704b422b334a8370d378a
paper_authors.name: Kaniuar Bacho
paper_authors._id: 679704b422b334a8370d378b
paper_authors.name: Pierre Clavier
paper_authors._id: 679704b422b334a8370d378c
paper_authors.name: Gabriel Recchia
paper_authors._id: 679704b422b334a8370d378d
paper_authors.name: Mara Popescu
paper_authors._id: 679704b422b334a8370d378e
paper_authors.name: Nikita Shulga
paper_authors._id: 679704b422b334a8370d378f
paper_authors.name: Ngefor Mildred Tanwie
paper_authors._id: 679704b422b334a8370d3790
paper_authors.name: Thomas C. H. Lux
paper_authors._id: 679704b422b334a8370d3791
paper_authors.name: Ben Rank
paper_authors._id: 679704b422b334a8370d3792
paper_authors.name: Colin Ni
paper_authors._id: 679704b422b334a8370d3793
paper_authors.name: Alesia Yakimchyk
paper_authors._id: 679704b422b334a8370d3794
paper_authors.name: Huanxu
paper_authors._id: 679704b422b334a8370d3795
paper_authors.name: Liu
paper_authors._id: 679704b422b334a8370d3796
paper_authors.name: Olle Häggström
paper_authors._id: 679704b422b334a8370d3797
paper_authors.name: Emil Verkama
paper_authors._id: 679704b422b334a8370d3798
paper_authors.name: Himanshu Narayan
paper_authors._id: 679704b422b334a8370d3799
paper_authors.name: Hans Gundlach
paper_authors._id: 679704b422b334a8370d379a
paper_authors.name: Leonor Brito-Santana
paper_authors._id: 679704b422b334a8370d379b
paper_authors.name: Brian Amaro
paper_authors._id: 679704b422b334a8370d379c
paper_authors.name: Vivek Vajipey
paper_authors._id: 679704b422b334a8370d379d
paper_authors.name: Rynaa Grover
paper_authors._id: 679704b422b334a8370d379e
paper_authors.name: Yiyang Fan
paper_authors._id: 679704b422b334a8370d379f
paper_authors.name: Gabriel Poesia Reis e Silva
paper_authors._id: 679704b422b334a8370d37a0
paper_authors.name: Linwei Xin
paper_authors._id: 679704b422b334a8370d37a1
paper_authors.name: Yosi Kratish
paper_authors._id: 679704b422b334a8370d37a2
paper_authors.name: Jakub Łucki
paper_authors._id: 679704b422b334a8370d37a3
paper_authors.name: Wen-Ding Li
paper_authors._id: 679704b422b334a8370d37a4
paper_authors.name: Justin Xu
paper_authors._id: 679704b422b334a8370d37a5
paper_authors.name: Kevin Joseph Scaria
paper_authors._id: 679704b422b334a8370d37a6
paper_authors.name: Freddie Vargus
paper_authors._id: 679704b422b334a8370d37a7
paper_authors.name: Farzad Habibi
paper_authors._id: 679704b422b334a8370d37a8
paper_authors.name: Long
paper_authors._id: 679704b422b334a8370d37a9
paper_authors.name: Lian
paper_authors._id: 679704b422b334a8370d37aa
paper_authors.name: Emanuele Rodolà
paper_authors._id: 679704b422b334a8370d37ab
paper_authors.name: Jules Robins
paper_authors._id: 679704b422b334a8370d37ac
paper_authors.name: Vincent Cheng
paper_authors._id: 679704b422b334a8370d37ad
paper_authors.name: Declan Grabb
paper_authors._id: 679704b422b334a8370d37ae
paper_authors.name: Ida Bosio
paper_authors._id: 679704b422b334a8370d37af
paper_authors.name: Tony Fruhauff
paper_authors._id: 679704b422b334a8370d37b0
paper_authors.name: Ido Akov
paper_authors._id: 679704b422b334a8370d37b1
paper_authors.name: Eve J. Y. Lo
paper_authors._id: 679704b422b334a8370d37b2
paper_authors.name: Hao Qi
paper_authors._id: 679704b422b334a8370d37b3
paper_authors.name: Xi Jiang
paper_authors._id: 679704b422b334a8370d37b4
paper_authors.name: Ben Segev
paper_authors._id: 679704b422b334a8370d37b5
paper_authors.name: Jingxuan Fan
paper_authors._id: 679704b422b334a8370d37b6
paper_authors.name: Sarah Martinson
paper_authors._id: 679704b422b334a8370d37b7
paper_authors.name: Erik Y. Wang
paper_authors._id: 679704b422b334a8370d37b8
paper_authors.name: Kaylie Hausknecht
paper_authors._id: 679704b422b334a8370d37b9
paper_authors.name: Michael P. Brenner
paper_authors._id: 679704b422b334a8370d37ba
paper_authors.name: Mao Mao
paper_authors._id: 679704b422b334a8370d37bb
paper_authors.name: Yibo Jiang
paper_authors._id: 679704b422b334a8370d37bc
paper_authors.name: Xinyu Zhang
paper_authors._id: 679704b422b334a8370d37bd
paper_authors.name: David Avagian
paper_authors._id: 679704b422b334a8370d37be
paper_authors.name: Eshawn Jessica Scipio
paper_authors._id: 679704b422b334a8370d37bf
paper_authors.name: Muhammad Rehan Siddiqi
paper_authors._id: 679704b422b334a8370d37c0
paper_authors.name: Alon Ragoler
paper_authors._id: 679704b422b334a8370d37c1
paper_authors.name: Justin Tan
paper_authors._id: 679704b422b334a8370d37c2
paper_authors.name: Deepakkumar Patil
paper_authors._id: 679704b422b334a8370d37c3
paper_authors.name: Rebeka Plecnik
paper_authors._id: 679704b422b334a8370d37c4
paper_authors.name: Aaron Kirtland
paper_authors._id: 679704b422b334a8370d37c5
paper_authors.name: Roselynn Grace Montecillo
paper_authors._id: 679704b422b334a8370d37c6
paper_authors.name: Stephane Durand
paper_authors._id: 679704b422b334a8370d37c7
paper_authors.name: Omer Faruk Bodur
paper_authors._id: 679704b422b334a8370d37c8
paper_authors.name: Zahra Adoul
paper_authors._id: 679704b422b334a8370d37c9
paper_authors.name: Mohamed Zekry
paper_authors._id: 679704b422b334a8370d37ca
paper_authors.name: Guillaume Douville
paper_authors._id: 679704b422b334a8370d37cb
paper_authors.name: Ali Karakoc
paper_authors._id: 679704b422b334a8370d37cc
paper_authors.name: Tania C. B. Santos
paper_authors._id: 679704b422b334a8370d37cd
paper_authors.name: Samir Shamseldeen
paper_authors._id: 679704b422b334a8370d37ce
paper_authors.name: Loukmane Karim
paper_authors._id: 679704b422b334a8370d37cf
paper_authors.name: Anna Liakhovitskaia
paper_authors._id: 679704b422b334a8370d37d0
paper_authors.name: Nate Resman
paper_authors._id: 679704b422b334a8370d37d1
paper_authors.name: Nicholas Farina
paper_authors._id: 679704b422b334a8370d37d2
paper_authors.name: Juan Carlos Gonzalez
paper_authors._id: 679704b422b334a8370d37d3
paper_authors.name: Gabe Maayan
paper_authors._id: 679704b422b334a8370d37d4
paper_authors.name: Sarah Hoback
paper_authors._id: 679704b422b334a8370d37d5
paper_authors.name: Rodrigo De Oliveira Pena
paper_authors._id: 679704b422b334a8370d37d6
paper_authors.name: Glen Sherman
paper_authors._id: 679704b422b334a8370d37d7
paper_authors.name: Hodjat Mariji
paper_authors._id: 679704b422b334a8370d37d8
paper_authors.name: Rasoul Pouriamanesh
paper_authors._id: 679704b422b334a8370d37d9
paper_authors.name: Wentao Wu
paper_authors._id: 679704b422b334a8370d37da
paper_authors.name: Gözdenur Demir
paper_authors._id: 679704b422b334a8370d37db
paper_authors.name: Sandra Mendoza
paper_authors._id: 679704b422b334a8370d37dc
paper_authors.name: Ismail Alarab
paper_authors._id: 679704b422b334a8370d37dd
paper_authors.name: Joshua Cole
paper_authors._id: 679704b422b334a8370d37de
paper_authors.name: Danyelle Ferreira
paper_authors._id: 679704b422b334a8370d37df
paper_authors.name: Bryan Johnson
paper_authors._id: 679704b422b334a8370d37e0
paper_authors.name: Hsiaoyun Milliron
paper_authors._id: 679704b422b334a8370d37e1
paper_authors.name: Mohammad Safdari
paper_authors._id: 679704b422b334a8370d37e2
paper_authors.name: Liangti Dai
paper_authors._id: 679704b422b334a8370d37e3
paper_authors.name: Siriphan Arthornthurasuk
paper_authors._id: 679704b422b334a8370d37e4
paper_authors.name: Alexey Pronin
paper_authors._id: 679704b422b334a8370d37e5
paper_authors.name: Jing Fan
paper_authors._id: 679704b422b334a8370d37e6
paper_authors.name: Angel Ramirez-Trinidad
paper_authors._id: 679704b422b334a8370d37e7
paper_authors.name: Ashley Cartwright
paper_authors._id: 679704b422b334a8370d37e8
paper_authors.name: Daphiny Pottmaier
paper_authors._id: 679704b422b334a8370d37e9
paper_authors.name: Omid Taheri
paper_authors._id: 679704b422b334a8370d37ea
paper_authors.name: David Outevsky
paper_authors._id: 679704b422b334a8370d37eb
paper_authors.name: Stanley Stepanic
paper_authors._id: 679704b422b334a8370d37ec
paper_authors.name: Samuel Perry
paper_authors._id: 679704b422b334a8370d37ed
paper_authors.name: Luke Askew
paper_authors._id: 679704b422b334a8370d37ee
paper_authors.name: Raúl Adrián Huerta Rodríguez
paper_authors._id: 679704b422b334a8370d37ef
paper_authors.name: Abdelkader Dendane
paper_authors._id: 679704b422b334a8370d37f0
paper_authors.name: Sam Ali
paper_authors._id: 679704b422b334a8370d37f1
paper_authors.name: Ricardo Lorena
paper_authors._id: 679704b422b334a8370d37f2
paper_authors.name: Krishnamurthy Iyer
paper_authors._id: 679704b422b334a8370d37f3
paper_authors.name: Sk Md Salauddin
paper_authors._id: 679704b422b334a8370d37f4
paper_authors.name: Murat Islam
paper_authors._id: 679704b422b334a8370d37f5
paper_authors.name: Juan Gonzalez
paper_authors._id: 679704b422b334a8370d37f6
paper_authors.name: Josh Ducey
paper_authors._id: 679704b422b334a8370d37f7
paper_authors.name: Russell Campbell
paper_authors._id: 679704b422b334a8370d37f8
paper_authors.name: Maja Somrak
paper_authors._id: 679704b422b334a8370d37f9
paper_authors.name: Vasilios Mavroudis
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-03T08:16:07.872Z
paper_authors._id: 6394e3bb0663665c5f01f688
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1670855654319-6394e3bb0663665c5f01f688.jpeg
paper_authors.fullname: Vasilios Mavroudis
paper_authors.type: user
paper_authors.user: vasilisM
paper_authors._id: 679704b422b334a8370d37fa
paper_authors.name: Eric Vergo
paper_authors._id: 679704b422b334a8370d37fb
paper_authors.name: Juehang Qin
paper_authors._id: 679704b422b334a8370d37fc
paper_authors.name: Benjámin Borbás
paper_authors._id: 679704b422b334a8370d37fd
paper_authors.name: Eric Chu
paper_authors._id: 679704b422b334a8370d37fe
paper_authors.name: Jack Lindsey
paper_authors._id: 679704b422b334a8370d37ff
paper_authors.name: Anil Radhakrishnan
paper_authors._id: 679704b422b334a8370d3800
paper_authors.name: Antoine Jallon
paper_authors._id: 679704b422b334a8370d3801
paper_authors.name: I. M. J. McInnis
paper_authors._id: 679704b422b334a8370d3802
paper_authors.name: Alex Hoover
paper_authors._id: 679704b422b334a8370d3803
paper_authors.name: Sören Möller
paper_authors._id: 679704b422b334a8370d3804
paper_authors.name: Song Bian
paper_authors._id: 679704b422b334a8370d3805
paper_authors.name: John Lai
paper_authors._id: 679704b422b334a8370d3806
paper_authors.name: Tejal Patwardhan
paper_authors._id: 679704b422b334a8370d3807
paper_authors.name: Summer Yue
paper_authors._id: 679704b422b334a8370d3808
paper_authors.name: Alexandr Wang
paper_authors._id: 679704b422b334a8370d3809
paper_authors.name: Dan Hendrycks
paper_title: Humanity's Last Exam
paper_summary: Benchmarks are important tools for tracking the rapid advancements in large
  language model (LLM) capabilities. However, benchmarks are not keeping pace in
  difficulty: LLMs now achieve over 90\% accuracy on popular benchmarks like
  MMLU, limiting informed measurement of state-of-the-art LLM capabilities. In
  response, we introduce Humanity's Last Exam (HLE), a multi-modal benchmark at
  the frontier of human knowledge, designed to be the final closed-ended academic
  benchmark of its kind with broad subject coverage. HLE consists of 3,000
  questions across dozens of subjects, including mathematics, humanities, and the
  natural sciences. HLE is developed globally by subject-matter experts and
  consists of multiple-choice and short-answer questions suitable for automated
  grading. Each question has a known solution that is unambiguous and easily
  verifiable, but cannot be quickly answered via internet retrieval.
  State-of-the-art LLMs demonstrate low accuracy and calibration on HLE,
  highlighting a significant gap between current LLM capabilities and the expert
  human frontier on closed-ended academic questions. To inform research and
  policymaking upon a clear understanding of model capabilities, we publicly
  release HLE at https://lastexam.ai.
paper_ai_keywords: large language model (LLM) benchmarks modal multilingual benchmark MMLU decoding multi-modal benchmark human knowledge academia closed-ended human knowledge subject coverage multiple-choice short-answer automated grading known solution ambiguity verifiability internet retrieval state-of-the-art accuracy calibration expert model capabilities research policymaking clear understanding