Architecture Decoupling Is Not All You Need For Unified Multimodal Model
Paper
โข
2511.22663
โข
Published
โข
29
Nitesh Kumar Sharma
carbene101
AI & ML interests
LLMs, OCR
Recent Activity
upvoted
a
paper
about 1 month ago
Architecture Decoupling Is Not All You Need For Unified Multimodal Model
reacted
to
sergiopaniego's
post
with ๐ฅ
about 1 month ago
we've just added several example scripts to TRL showing how to train models with GRPO using some of the new OpenEnv environments
train a model to interact with a browser (๐ฎ BrowserGym Env), play Wordle (๐ฎ Wordle Env) and moooore!
TRL (GRPO + vLLM) + OpenEnv! โก๏ธ
๐ go play with them: https://github.com/huggingface/trl/tree/main/examples/scripts/openenv
๐ examples list: https://huggingface.co/docs/trl/main/en/example_overview#scripts
Organizations
upvoted
a
paper
about 1 month ago
reacted to
sergiopaniego's
post with ๐ฅ
about 1 month ago
Post
2594
we've just added several example scripts to TRL showing how to train models with GRPO using some of the new OpenEnv environments
train a model to interact with a browser (๐ฎ BrowserGym Env), play Wordle (๐ฎ Wordle Env) and moooore!
TRL (GRPO + vLLM) + OpenEnv! โก๏ธ
๐ go play with them: https://github.com/huggingface/trl/tree/main/examples/scripts/openenv
๐ examples list: https://huggingface.co/docs/trl/main/en/example_overview#scripts
train a model to interact with a browser (๐ฎ BrowserGym Env), play Wordle (๐ฎ Wordle Env) and moooore!
TRL (GRPO + vLLM) + OpenEnv! โก๏ธ
๐ go play with them: https://github.com/huggingface/trl/tree/main/examples/scripts/openenv
๐ examples list: https://huggingface.co/docs/trl/main/en/example_overview#scripts
reacted to
Kseniase's
post with โค๏ธ
about 2 months ago
Post
6113
12 Types of JEPA
Since Yann LeCun together with Randall Balestriero released a new paper on JEPA (Joint-Embedding Predictive Architecture), laying out its theory and introducing an efficient practical version called LeJEPA, we figured you might need even more JEPA. Here are 7 recent JEPA variants plus 5 iconic ones:
1. LeJEPA โ LeJEPA: Provable and Scalable Self-Supervised Learning Without the Heuristics (2511.08544)
Explains a full theory for JEPAs, defining the โidealโ JEPA embedding as an isotropic Gaussian, and proposes the SIGReg objective to push JEPA toward this ideal, resulting in practical LeJEPA
2. JEPA-T โ JEPA-T: Joint-Embedding Predictive Architecture with Text Fusion for Image Generation (2510.00974)
A text-to-image model that tokenizes images and captions with a joint predictive Transformer, enhances fusion with cross-attention and text embeddings before training loss, and generates images by iteratively denoising visual tokens conditioned on text
3. Text-JEPA โ Speaking in Words, Thinking in Logic: A Dual-Process Framework in QA Systems (2507.20491)
Converts natural language into first-order logic, with a Z3 solver handling reasoning, enabling efficient, explainable QA with far lower compute than large LLMs
4. N-JEPA (Noise-based JEPA) โ Improving Joint Embedding Predictive Architecture with Diffusion Noise (2507.15216)
Connects self-supervised learning with diffusion-style noise by using noise-based masking and multi-level schedules, especially improving visual classification
5. SparseJEPA โ SparseJEPA: Sparse Representation Learning of Joint Embedding Predictive Architectures (2504.16140)
Adds sparse representation learning to make embeddings more interpretable and efficient. It groups latent variables by shared semantic structure using a sparsity penalty while preserving accuracy
6. TS-JEPA (Time Series JEPA) โ Joint Embeddings Go Temporal (2509.25449)
Adapts JEPA to time-series by learning latent self-supervised representations and predicting future latents for robustness to noise and confounders
Read further below โ
It you like it, also subscribe to the Turing Post: https://www.turingpost.com/subscribe
Since Yann LeCun together with Randall Balestriero released a new paper on JEPA (Joint-Embedding Predictive Architecture), laying out its theory and introducing an efficient practical version called LeJEPA, we figured you might need even more JEPA. Here are 7 recent JEPA variants plus 5 iconic ones:
1. LeJEPA โ LeJEPA: Provable and Scalable Self-Supervised Learning Without the Heuristics (2511.08544)
Explains a full theory for JEPAs, defining the โidealโ JEPA embedding as an isotropic Gaussian, and proposes the SIGReg objective to push JEPA toward this ideal, resulting in practical LeJEPA
2. JEPA-T โ JEPA-T: Joint-Embedding Predictive Architecture with Text Fusion for Image Generation (2510.00974)
A text-to-image model that tokenizes images and captions with a joint predictive Transformer, enhances fusion with cross-attention and text embeddings before training loss, and generates images by iteratively denoising visual tokens conditioned on text
3. Text-JEPA โ Speaking in Words, Thinking in Logic: A Dual-Process Framework in QA Systems (2507.20491)
Converts natural language into first-order logic, with a Z3 solver handling reasoning, enabling efficient, explainable QA with far lower compute than large LLMs
4. N-JEPA (Noise-based JEPA) โ Improving Joint Embedding Predictive Architecture with Diffusion Noise (2507.15216)
Connects self-supervised learning with diffusion-style noise by using noise-based masking and multi-level schedules, especially improving visual classification
5. SparseJEPA โ SparseJEPA: Sparse Representation Learning of Joint Embedding Predictive Architectures (2504.16140)
Adds sparse representation learning to make embeddings more interpretable and efficient. It groups latent variables by shared semantic structure using a sparsity penalty while preserving accuracy
6. TS-JEPA (Time Series JEPA) โ Joint Embeddings Go Temporal (2509.25449)
Adapts JEPA to time-series by learning latent self-supervised representations and predicting future latents for robustness to noise and confounders
Read further below โ
It you like it, also subscribe to the Turing Post: https://www.turingpost.com/subscribe
- 1 reply
upvoted
an
article
2 months ago
Article
The 1 Billion Token Challenge: Finding the Perfect Pre-training Mix
โข
53
`Qwen3VLModel` object has no attribute 'generate'
โ
1
6
#1 opened 2 months ago
by
carbene101
liked
a
model
2 months ago
reacted to
tomaarsen's
post with ๐
2 months ago
Post
4329
๐ค Sentence Transformers is joining Hugging Face! ๐ค This formalizes the existing maintenance structure, as I've personally led the project for the past two years on behalf of Hugging Face! Details:
Today, the Ubiquitous Knowledge Processing (UKP) Lab is transferring the project to Hugging Face. Sentence Transformers will remain a community-driven, open-source project, with the same open-source license (Apache 2.0) as before. Contributions from researchers, developers, and enthusiasts are welcome and encouraged. The project will continue to prioritize transparency, collaboration, and broad accessibility.
Read our full announcement for more details and quotes from UKP and Hugging Face leadership: https://huggingface.co/blog/sentence-transformers-joins-hf
We see an increasing wish from companies to move from large LLM APIs to local models for better control and privacy, reflected in the library's growth: in just the last 30 days, Sentence Transformer models have been downloaded >270 million times, second only to transformers.
I would like to thank the UKP Lab, and especially Nils Reimers and Iryna Gurevych, both for their dedication to the project and for their trust in myself, both now and two years ago. Back then, neither of you knew me well, yet you trusted me to take the project to new heights. That choice ended up being very valuable for the embedding & Information Retrieval community, and I think this choice of granting Hugging Face stewardship will be similarly successful.
I'm very excited about the future of the project, and for the world of embeddings and retrieval at large!
Today, the Ubiquitous Knowledge Processing (UKP) Lab is transferring the project to Hugging Face. Sentence Transformers will remain a community-driven, open-source project, with the same open-source license (Apache 2.0) as before. Contributions from researchers, developers, and enthusiasts are welcome and encouraged. The project will continue to prioritize transparency, collaboration, and broad accessibility.
Read our full announcement for more details and quotes from UKP and Hugging Face leadership: https://huggingface.co/blog/sentence-transformers-joins-hf
We see an increasing wish from companies to move from large LLM APIs to local models for better control and privacy, reflected in the library's growth: in just the last 30 days, Sentence Transformer models have been downloaded >270 million times, second only to transformers.
I would like to thank the UKP Lab, and especially Nils Reimers and Iryna Gurevych, both for their dedication to the project and for their trust in myself, both now and two years ago. Back then, neither of you knew me well, yet you trusted me to take the project to new heights. That choice ended up being very valuable for the embedding & Information Retrieval community, and I think this choice of granting Hugging Face stewardship will be similarly successful.
I'm very excited about the future of the project, and for the world of embeddings and retrieval at large!
- 1 reply
reacted to
appvoid's
post with ๐
2 months ago
Post
4096
today is going to be a great day for small models, are you ready?
- 3 replies
reacted to
SelmaNajih001's
post with ๐
3 months ago
Post
3401
Introducing a Hugging Face Tutorial on Regression
While Hugging Face offers extensive tutorials on classification and NLP tasks, there is very little guidance on performing regression tasks with Transformers.
In my latest article, I provide a step-by-step guide to running regression using Hugging Face, applying it to financial news data to predict stock returns.
In this tutorial, you will learn how to:
-Prepare and preprocess textual and numerical data for regression
-Configure a Transformer model for regression tasks
-Apply the model to real-world financial datasets with fully reproducible code
Read the full article here: https://huggingface.co/blog/SelmaNajih001/how-to-run-a-regression-using-hugging-face
The dataset used: SelmaNajih001/FinancialClassification
While Hugging Face offers extensive tutorials on classification and NLP tasks, there is very little guidance on performing regression tasks with Transformers.
In my latest article, I provide a step-by-step guide to running regression using Hugging Face, applying it to financial news data to predict stock returns.
In this tutorial, you will learn how to:
-Prepare and preprocess textual and numerical data for regression
-Configure a Transformer model for regression tasks
-Apply the model to real-world financial datasets with fully reproducible code
Read the full article here: https://huggingface.co/blog/SelmaNajih001/how-to-run-a-regression-using-hugging-face
The dataset used: SelmaNajih001/FinancialClassification
- 1 reply
reacted to
sergiopaniego's
post with ๐ฅ
3 months ago
Post
3494
๐ฅ Tons of new material just landed in the smol-course! ๐งโ๐ป
> evaluation
> alignment
> VLMs
> quizzes
> assignments!
> certificates!๐ฉโ๐
go learn! ๐ https://huggingface.co/learn/smol-course/unit0/1
> evaluation
> alignment
> VLMs
> quizzes
> assignments!
> certificates!๐ฉโ๐
go learn! ๐ https://huggingface.co/learn/smol-course/unit0/1
- 1 reply
upvoted
an
article
3 months ago
Article
Smol2Operator: Post-Training GUI Agents for Computer Use
- +3
โข
134
upvoted
an
article
4 months ago
Article
Unleashing the Full Potential of ERNIE4.5 using FastDeploy
โข
11
Were the original PDFs saved?
11
#2 opened 4 months ago
by
staghado
commented on
Exploring Environments Hub: Your Language Model needs better (open) environments to learn
4 months ago
Thanks for sharing this!
upvoted
an
article
4 months ago
Article
NVIDIA Releases 3 Million Sample Dataset for OCR, Visual Question Answering, and Captioning Tasks
โข
75
reacted to
codelion's
post with ๐ฅ
4 months ago
Post
4958
I recently added a recipe in ellora to improve reasoning capabilities to Gemma-3-1B using self-supervised learning. Model now shows step-by-step thinking in <think> tags before answering.
Logic puzzle accuracy: 61% โ 84%. 3 hours training on single GPU. ๐ง
Used GRPO where model generates multiple responses and learns to prefer better reasoning. Works surprisingly well for making smaller models more transparent.
๐ Colab: https://colab.research.google.com/github/codelion/ellora/blob/main/Ellora_Recipe_2_Reasoning_LoRA_with_Self-Rewarding_GRPO.ipynb
๐ค Model: codelion/gemma-3-1b-it-reasoning-grpo-lora
๐ป Code: https://github.com/codelion/ellora
Logic puzzle accuracy: 61% โ 84%. 3 hours training on single GPU. ๐ง
Used GRPO where model generates multiple responses and learns to prefer better reasoning. Works surprisingly well for making smaller models more transparent.
๐ Colab: https://colab.research.google.com/github/codelion/ellora/blob/main/Ellora_Recipe_2_Reasoning_LoRA_with_Self-Rewarding_GRPO.ipynb
๐ค Model: codelion/gemma-3-1b-it-reasoning-grpo-lora
๐ป Code: https://github.com/codelion/ellora
- 1 reply
upvoted
an
article
6 months ago
Article
Ettin Suite: SoTA Paired Encoders and Decoders
- +4
โข
77
commented on
SmolLM3: smol, multilingual, long-context reasoner
6 months ago
Wow, great going and amazing writeup!
reacted to
sergiopaniego's
post with ๐ฅ
6 months ago
Post
1078
๐ฃ CALL FOR CONTRIBUTORS! ๐ฃ
Following last weekโs full release of Gemma 3n, we launched a dedicated recipes repo to explore and share use cases. We already added some! ๐งโ๐ณ
Now weโre inviting the community to contribute and showcase how these models shine! โจ
Let them cook.
Check it out: https://github.com/huggingface/huggingface-gemma-recipes/issues/4
Following last weekโs full release of Gemma 3n, we launched a dedicated recipes repo to explore and share use cases. We already added some! ๐งโ๐ณ
Now weโre inviting the community to contribute and showcase how these models shine! โจ
Let them cook.
Check it out: https://github.com/huggingface/huggingface-gemma-recipes/issues/4
- 1 reply
liked
a
Space
8 months ago