Improving Hugging Face Training Efficiency Through Packing with Flash Attention
•
25
Mayank Mishra
mayank-mishra
AI & ML interests
Large Language Models, Distributed Training and Inference
Recent Activity
upvoted
a
collection
6 days ago
Granite 3.1 Language Models
new activity
6 days ago
ibm-granite/granite-3.1-8b-instruct:Exceptional creative writer
authored
a paper
6 days ago
Selective Self-Rehearsal: A Fine-Tuning Approach to Improve
Generalization in Large Language Models
Articles
Organizations
Posts
4
Post
1781
New preprint out with colleagues from MIT and IBM Research
Reducing Transformer Key-Value Cache Size with Cross-Layer Attention (2405.12981)
We introduce a simple mechanism of sharing keys and values across layers, reducing the memory needed for KV cache during inference!!
Reducing Transformer Key-Value Cache Size with Cross-Layer Attention (2405.12981)
We introduce a simple mechanism of sharing keys and values across layers, reducing the memory needed for KV cache during inference!!
Post
2560
Thrilled to unveil DS-MoE: a dense training and sparse inference scheme for enhanced computational and memory efficiency in your MoE models! 🚀🚀🚀
Discover more in our blog: https://huggingface.co/blog/bpan/ds-moe and dive into the details with our paper: Dense Training, Sparse Inference: Rethinking Training of Mixture-of-Experts Language Models (2404.05567)
Discover more in our blog: https://huggingface.co/blog/bpan/ds-moe and dive into the details with our paper: Dense Training, Sparse Inference: Rethinking Training of Mixture-of-Experts Language Models (2404.05567)
Collections
1
models
8
mayank-mishra/granite-3b-code-glaive-20k
Text Generation
•
Updated
•
25
mayank-mishra/granite-20b-code-instruct-Q4_K_M-GGUF
Text Generation
•
Updated
•
2
mayank-mishra/starcoder-GPTQ-8bit-128g
Updated
•
11
mayank-mishra/starcoder-GPTQ-4bit-128g
Updated
•
16
mayank-mishra/starcoderbase-GPTQ-4bit-128g
Updated
•
21
mayank-mishra/starcoderbase-GPTQ-8bit-128g
Updated
•
3
mayank-mishra/santacoder-GPTQ-4bit-128g
Updated
•
2
mayank-mishra/santacoder-GPTQ-8bit-128g
Updated
•
1