Continual Quantization-Aware Pre-Training: When to transition from 16-bit to 1.58-bit pre-training for BitNet language models?
Paper
•
2502.11895
•
Published
•
2
Lukas Galke
lgalke
·
AI & ML interests
Representation learning; pre-training; quantization; continual learning; probing; multilingual pre-training; knowledge tracing/discovering/inducing
Recent Activity
View all activity
Organizations
authored
6
papers
26 days ago
What makes a language easy to deep-learn? Deep neural networks and humans similarly benefit from compositional structure
Paper
•
2302.12239
•
Published
•
1
Dynaword: From One-shot to Continuously Developed Datasets
Paper
•
2508.02271
•
Published
•
13
GenCodeSearchNet: A Benchmark Test Suite for Evaluating Generalization in Programming Language Understanding
Paper
•
2311.09707
•
Published
When are 1.58 bits enough? A Bottom-up Exploration of BitNet Quantization
Paper
•
2411.05882
•
Published
•
1
CBOW Is Not All You Need: Combining CBOW with the Compositional Matrix Space Model
Paper
•
1902.06423
•
Published
upvoted
a
paper
28 days ago
liked
2
models
5 months ago
liked
2
Spaces
5 months ago
liked
3
models
5 months ago
liked
3
datasets
5 months ago
upvoted
an
article
6 months ago
Article
Finally, a Replacement for BERT: Introducing ModernBERT
By
and 14 others
•
•
683
liked
a
dataset
6 months ago
liked
a
model
7 months ago
