Toward Evaluative Thinking: Meta Policy Optimization with Evolving Reward Models
Paper
•
2504.20157
•
Published
•
38
AI & ML interests
Contributors who are invited to beta-test our next big feature! Contact us if you want to join this team :-)
Recent Activity
View all activity
machineteacher
authored
a
paper
3 months ago
mohsenfayyaz
authored
a
paper
5 months ago
fuwei
authored
a
paper
5 months ago
Post
744
New state-of-the-art BERT-size retrieval model: *cde-small-v2* 🥳🍾
Hi everyone! We at Cornell are releasing a new retrieval model this week. It uses the contextual embeddings framework, is based on ModernBERT backbone, and gets state-of-the-art results on the MTEB benchmark for its model size (140M parameters). cde-small-v2 gets an average score of 65.6 across the 56 datasets and sees improvements from our previous model in *every* task domain (retrieval, classification, etc.).
We made a lot of changes to make this model work. First of all, ModernBERT has a better tokenizer, which probably helped this work out-of-the-box. We also followed the principles from the CDE paper and used harder clusters and better hard-negative filtering, which showed a small performance improvement. And we made a few small changes that have been shown to work on the larger models: we disabled weight decay, masked out the prefix tokens during pooling, and added a residual connection from the first-stage to the second-stage for better gradient flow.
We're still looking for a computer sponsor to help us scale CDE to larger models. Since it's now state-of-the-art at the 100M parameter scale, it seems to be a reasonable bet that we could train a state-of-the-art large model if we had the GPUs. If you're interested in helping with this, please reach out!
Here's a link to the model: jxm/cde-small-v2
And here's a link to the paper: Contextual Document Embeddings (2410.02525)
Hi everyone! We at Cornell are releasing a new retrieval model this week. It uses the contextual embeddings framework, is based on ModernBERT backbone, and gets state-of-the-art results on the MTEB benchmark for its model size (140M parameters). cde-small-v2 gets an average score of 65.6 across the 56 datasets and sees improvements from our previous model in *every* task domain (retrieval, classification, etc.).
We made a lot of changes to make this model work. First of all, ModernBERT has a better tokenizer, which probably helped this work out-of-the-box. We also followed the principles from the CDE paper and used harder clusters and better hard-negative filtering, which showed a small performance improvement. And we made a few small changes that have been shown to work on the larger models: we disabled weight decay, masked out the prefix tokens during pooling, and added a residual connection from the first-stage to the second-stage for better gradient flow.
We're still looking for a computer sponsor to help us scale CDE to larger models. Since it's now state-of-the-art at the 100M parameter scale, it seems to be a reasonable bet that we could train a state-of-the-art large model if we had the GPUs. If you're interested in helping with this, please reach out!
Here's a link to the model: jxm/cde-small-v2
And here's a link to the paper: Contextual Document Embeddings (2410.02525)
bhoov
authored
3
papers
12 months ago
DiffusionDB: A Large-scale Prompt Gallery Dataset for Text-to-Image Generative Models
Paper
•
2210.14896
•
Published
•
1
Diffusion Explainer: Visual Explanation for Text-to-image Stable Diffusion
Paper
•
2305.03509
•
Published
•
1
Transformer Explainer: Interactive Learning of Text-Generative Models
Paper
•
2408.04619
•
Published
•
161
huseinzol05
authored
3
papers
over 1 year ago
Multi-Lingual Malaysian Embedding: Leveraging Large Language Models for Semantic Representations
Paper
•
2402.03053
•
Published
•
2
MaLLaM -- Malaysia Large Language Model
Paper
•
2401.14680
•
Published
•
3
Large Malaysian Language Model Based on Mistral for Enhanced Local Language Understanding
Paper
•
2401.13565
•
Published
•
4
isaaccorley
authored
8
papers
over 1 year ago
Revisiting pre-trained remote sensing model benchmarks: resizing and normalization matters
Paper
•
2305.13456
•
Published
TorchGeo: Deep Learning With Geospatial Data
Paper
•
2111.08872
•
Published
•
1
Destruction of Image Steganography using Generative Adversarial Networks
Paper
•
1912.10070
•
Published
Supervising Remote Sensing Change Detection Models with 3D Surface Semantics
Paper
•
2202.13251
•
Published
DomainGAN: Generating Adversarial Examples to Attack Domain Generation Algorithm Classifiers
Paper
•
1911.06285
•
Published
SSL4EO-L: Datasets and Foundation Models for Landsat Imagery
Paper
•
2306.09424
•
Published
Single-View Height Estimation with Conditional Diffusion Probabilistic Models
Paper
•
2304.13214
•
Published
ZRG: A Dataset for Multimodal 3D Residential Rooftop Understanding
Paper
•
2304.13219
•
Published
machineteacher
authored
a
paper
over 1 year ago
parker
authored
a
paper
about 2 years ago