Global MMLU: Understanding and Addressing Cultural and Linguistic Biases in Multilingual Evaluation
Paper
•
2412.03304
•
Published
•
17
AIM-Harvard
university
AI & ML interests
Artificial Intelligence in Medicine (AIM) Program (NLP group/Bitterman lab: https://www.bittermanlab.org/)
Recent Activity
View all activity
AIM-Harvard's activity
clefourrier
authored
a
paper
20 days ago
Luoyu8631
updated
a
dataset
about 1 month ago
shanchen
updated
a
dataset
about 1 month ago
shanchen
authored
a
paper
about 1 month ago
daniellebitt
authored
a
paper
about 1 month ago
shanchen
updated
a
dataset
about 2 months ago
shanchen
authored
2
papers
2 months ago
Wait, but Tylenol is Acetaminophen... Investigating and Improving Language Models' Ability to Resist Requests for Misinformation
Paper
•
2409.20385
•
Published
WorldMedQA-V: a multilingual, multimodal medical examination dataset for multimodal language models evaluation
Paper
•
2410.12722
•
Published
•
5
clefourrier
authored
2
papers
6 months ago
daniellebitt
authored
a
paper
6 months ago
mingye94
authored
a
paper
6 months ago
gallifantjack
authored
a
paper
6 months ago
NikolajMunch
authored
2
papers
6 months ago
shanchen
authored
3
papers
6 months ago
Language Models are Surprisingly Fragile to Drug Names in Biomedical Benchmarks
Paper
•
2406.12066
•
Published
•
8
Cross-Care: Assessing the Healthcare Implications of Pre-training Data on Language Model Bias
Paper
•
2405.05506
•
Published
•
1
Measuring Pointwise $\mathcal{V}$-Usable Information In-Context-ly
Paper
•
2310.12300
•
Published
•
1
clefourrier
posted
an
update
8 months ago
Post
5449
In a basic chatbots, errors are annoyances. In medical LLMs, errors can have life-threatening consequences 🩸
It's therefore vital to benchmark/follow advances in medical LLMs before even thinking about deployment.
This is why a small research team introduced a medical LLM leaderboard, to get reproducible and comparable results between LLMs, and allow everyone to follow advances in the field.
openlifescienceai/open_medical_llm_leaderboard
Congrats to @aaditya and @pminervini !
Learn more in the blog: https://huggingface.co/blog/leaderboard-medicalllm
It's therefore vital to benchmark/follow advances in medical LLMs before even thinking about deployment.
This is why a small research team introduced a medical LLM leaderboard, to get reproducible and comparable results between LLMs, and allow everyone to follow advances in the field.
openlifescienceai/open_medical_llm_leaderboard
Congrats to @aaditya and @pminervini !
Learn more in the blog: https://huggingface.co/blog/leaderboard-medicalllm
clefourrier
posted
an
update
8 months ago
Post
4430
Contamination free code evaluations with LiveCodeBench! 🖥️
LiveCodeBench is a new leaderboard, which contains:
- complete code evaluations (on code generation, self repair, code execution, tests)
- my favorite feature: problem selection by publication date 📅
This feature means that you can get model scores averaged only on new problems out of the training data. This means... contamination free code evals! 🚀
Check it out!
Blog: https://huggingface.co/blog/leaderboard-livecodebench
Leaderboard: livecodebench/leaderboard
Congrats to @StringChaos @minimario @xu3kev @kingh0730 and @FanjiaYan for the super cool leaderboard!
LiveCodeBench is a new leaderboard, which contains:
- complete code evaluations (on code generation, self repair, code execution, tests)
- my favorite feature: problem selection by publication date 📅
This feature means that you can get model scores averaged only on new problems out of the training data. This means... contamination free code evals! 🚀
Check it out!
Blog: https://huggingface.co/blog/leaderboard-livecodebench
Leaderboard: livecodebench/leaderboard
Congrats to @StringChaos @minimario @xu3kev @kingh0730 and @FanjiaYan for the super cool leaderboard!