BigCodeBench: Benchmarking Large Language Models on Solving Practical and Challenging Programming Tasks
•
47
BigCode
Enterprise
non-profit
AI & ML interests
None defined yet.
Recent Activity
View all activity
Articles
bigcode's activity
JeanKaddour
authored
a
paper
2 days ago
Veeru
authored
a
paper
3 days ago
gagan3012
authored
a
paper
10 days ago
Evaluation on local GPUs
1
#4 opened about 1 month ago
by
gauraviiitian
Dataset splits
1
#5 opened about 1 month ago
by
danfperam
terryyz
updated
a
dataset
about 1 month ago
hughesthe1st
updated
a
dataset
about 1 month ago
natolambert
authored
a
paper
about 1 month ago
mpatwary
authored
a
paper
about 1 month ago
terryyz
updated
8
datasets
about 2 months ago
bigcode/bigcodebench-hard-results
Viewer
•
Updated
•
202
•
50
•
1
bigcode/bigcodebench-hard-solve-rate
Viewer
•
Updated
•
296
•
52
•
1
bigcode/bigcodebench-hard-domain
Viewer
•
Updated
•
372
•
70
•
1
bigcode/bigcodebench-hard-perf
Viewer
•
Updated
•
372
•
18
bigcode/bigcodebench-results
Viewer
•
Updated
•
202
•
44
•
2
bigcode/bigcodebench-solve-rate
Viewer
•
Updated
•
2.28k
•
54
bigcode/bigcodebench-domain
Viewer
•
Updated
•
279
•
55
bigcode/bigcodebench-perf
Viewer
•
Updated
•
279
•
31
Post
3326
Today in Privacy & AI Tooling - introducing a nifty new tool to examine where data goes in open-source apps on 🤗
HF Spaces have tons (100Ks!) of cool demos leveraging or examining AI systems - and because most of them are OSS we can see exactly how they handle user data 📚🔍
That requires actually reading the code though, which isn't always easy or quick! Good news: code LMs have gotten pretty good at automatic review, so we can offload some of the work - here I'm using Qwen/Qwen2.5-Coder-32B-Instruct to generate reports and it works pretty OK 🙌
The app works in three stages:
1. Download all code files
2. Use the Code LM to generate a detailed report pointing to code where data is transferred/(AI-)processed (screen 1)
3. Summarize the app's main functionality and data journeys (screen 2)
4. Build a Privacy TLDR with those inputs
It comes with a bunch of pre-reviewed apps/Spaces, great to see how many process data locally or through (private) HF endpoints 🤗
Note that this is a POC, lots of exciting work to do to make it more robust, so:
- try it: yjernite/space-privacy
- reach out to collab: yjernite/space-privacy
HF Spaces have tons (100Ks!) of cool demos leveraging or examining AI systems - and because most of them are OSS we can see exactly how they handle user data 📚🔍
That requires actually reading the code though, which isn't always easy or quick! Good news: code LMs have gotten pretty good at automatic review, so we can offload some of the work - here I'm using Qwen/Qwen2.5-Coder-32B-Instruct to generate reports and it works pretty OK 🙌
The app works in three stages:
1. Download all code files
2. Use the Code LM to generate a detailed report pointing to code where data is transferred/(AI-)processed (screen 1)
3. Summarize the app's main functionality and data journeys (screen 2)
4. Build a Privacy TLDR with those inputs
It comes with a bunch of pre-reviewed apps/Spaces, great to see how many process data locally or through (private) HF endpoints 🤗
Note that this is a POC, lots of exciting work to do to make it more robust, so:
- try it: yjernite/space-privacy
- reach out to collab: yjernite/space-privacy
albanie
authored
a
paper
about 2 months ago
lvwerra
authored
a
paper
about 2 months ago