title: ModernBERT or DeBERTaV3? Examining Architecture and Data Influence on
    Transformer Encoder Models Performance
summary: Pretrained transformer-encoder models like DeBERTaV3 and ModernBERT introduce
  architectural advancements aimed at improving efficiency and performance.
  Although the authors of ModernBERT report improved performance over DeBERTaV3
  on several benchmarks, the lack of disclosed training data and the absence of
  comparisons using a shared dataset make it difficult to determine whether these
  gains are due to architectural improvements or differences in training data. In
  this work, we conduct a controlled study by pretraining ModernBERT on the same
  dataset as CamemBERTaV2, a DeBERTaV3 French model, isolating the effect of
  model design. Our results show that the previous model generation remains
  superior in sample efficiency and overall benchmark performance, with
  ModernBERT's primary advantage being faster training and inference speed.
  However, the new proposed model still provides meaningful architectural
  improvements compared to earlier models such as BERT and RoBERTa. Additionally,
  we observe that high-quality pre-training data accelerates convergence but does
  not significantly improve final performance, suggesting potential benchmark
  saturation. These findings show the importance of disentangling pretraining
  data from architectural innovations when evaluating transformer models.
paper_authors._id: 67fca0ca05cd5b5035123b7e
paper_authors.name: Wissam Antoun
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-14T18:40:06.158Z
paper_authors._id: 605350699008a64d7b43b78d
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1616431735085-605350699008a64d7b43b78d.jpeg
paper_authors.fullname: Wissam Antoun
paper_authors.type: user
paper_authors.user: wissamantoun
paper_authors._id: 67fca0ca05cd5b5035123b7f
paper_authors.name: Benoît Sagot
paper_authors._id: 67fca0ca05cd5b5035123b80
paper_authors.name: Djamé Seddah
paper_title: ModernBERT or DeBERTaV3? Examining Architecture and Data Influence on
    Transformer Encoder Models Performance
paper_summary: Pretrained transformer-encoder models like DeBERTaV3 and ModernBERT introduce
  architectural advancements aimed at improving efficiency and performance.
  Although the authors of ModernBERT report improved performance over DeBERTaV3
  on several benchmarks, the lack of disclosed training data and the absence of
  comparisons using a shared dataset make it difficult to determine whether these
  gains are due to architectural improvements or differences in training data. In
  this work, we conduct a controlled study by pretraining ModernBERT on the same
  dataset as CamemBERTaV2, a DeBERTaV3 French model, isolating the effect of
  model design. Our results show that the previous model generation remains
  superior in sample efficiency and overall benchmark performance, with
  ModernBERT's primary advantage being faster training and inference speed.
  However, the new proposed model still provides meaningful architectural
  improvements compared to earlier models such as BERT and RoBERTa. Additionally,
  we observe that high-quality pre-training data accelerates convergence but does
  not significantly improve final performance, suggesting potential benchmark
  saturation. These findings show the importance of disentangling pretraining
  data from architectural innovations when evaluating transformer models.
paper_ai_summary: A controlled study comparing ModernBERT with CamemBERTaV2 demonstrates that the original model design remains superior in sample efficiency and benchmark performance, though ModernBERT offers faster training and inference.
paper_ai_keywords: transformer-encoder models DeBERTaV3 ModernBERT CamemBERTaV2 sample efficiency benchmark performance pretraining data architectural innovations transformer models BERT RoBERTa high-quality pre-training data convergence benchmark saturation