legaltextai/modernbert-embed-ft-const-legal-matryoshka

ModernBERT Embed base LegalTextAI Matryoshka

This is a sentence-transformers model finetuned from nomic-ai/modernbert-embed-base on the json dataset. It maps sentences & paragraphs to a 768-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

Model Details

Model Description

• Model Type: Sentence Transformer
• Base model: nomic-ai/modernbert-embed-base
• Maximum Sequence Length: 8192 tokens
• Output Dimensionality: 768 dimensions
• Similarity Function: Cosine Similarity
• Training Dataset:
  • json
• Language: en
• License: apache-2.0

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 8192, 'do_lower_case': False}) with Transformer model: ModernBertModel 
  (1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
  (2): Normalize()
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("legaltextai/modernbert-embed-ft-const-legal-matryoshka")
# Run inference
sentences = [
    "Based on the court's ruling, what are the implications of Title VII regarding discrimination against employees based on their transgender status or failure to conform to sex stereotypes?",
    'Thus, even if we\xa0agreed with the Funeral Home that Rost\'s religious exercise would be substantially burdened by enforcing Title VII in this case, we would nevertheless REVERSE the district court\'s grant of summary judgment to the Funeral Home and hold instead that requiring the Funeral Home to comply with Title VII constitutes the least restrictive means of furthering the government\'s compelling interest in eradicating discrimination against Stephens on the basis of sex. Thus, even assuming Rost\'s religious exercise is substantially burdened by the EEOC\'s enforcement action in this case, we GRANT summary judgment to the EEOC on the Funeral Home\'s RFRA defense on this alternative ground.\n\n\xa0\n\n[ … ]\n\n[ … ]\n\n\xa0\n\nIII. CONCLUSION\n\nDiscrimination against employees, either because of their failure to conform to sex stereotypes or their transgender and transitioning status, is illegal under Title VII. The unrefuted facts show that the Funeral Home fired Stephens because she refused to abide by her employer\'s stereotypical conception of her sex, and therefore the EEOC is entitled to summary judgment as to its unlawful-termination claim. RFRA provides the Funeral Home with no relief because continuing to employ Stephens would not, as a matter of law, substantially burden Rost\'s religious exercise, and even if it did, the EEOC has shown that enforcing Title VII here is the least restrictive means of furthering its compelling interest in combating and eradicating sex discrimination. We therefore REVERSE the district court\'s grant of summary judgment in favor of the Funeral Home and GRANT summary judgment to the EEOC on its unlawful-termination claim. We also REVERSE the district court\'s grant of summary judgment on the EEOC\'s discriminatory-clothing-allowance claim, as the district court erred in failing to consider the EEOC\'s claim on the merits. We REMAND this case to the district court for further proceedings consistent with this opinion.\n\n[1]\xa0We refer to Stephens using female pronouns, in accordance with the preference she has expressed through her briefing to this court.\n\n[2]\xa0All facts drawn from Def.\'s Statement of Facts (R. 55) are undisputed.\xa0See\xa0R. 64 (Pl.\'s Counter Statement of Disputed Facts) (Page ID #2066-88).\n\n[3]\xa0See also\xa0Appellee Br. at 16 ("It is a helpful exercise to think about\xa0Price Waterhouse\xa0and imagine that there was a dress code imposed which obligated Ms. Hopkins to wear a skirt while her male colleagues were obliged to wear pants. Had she simply been fired for wearing pants rather than a skirt, the case would have ended there — both sexes would have been equally burdened by the requirement to comply with their respective sex-specific standard. But what the firm could not do was fire her for being aggressive or macho when it was tolerating or rewarding the behavior among men — and when it did, it relied on a stereotype to treat her disparately from the men in the firm.").\n\n[4]\xa0Moreover, discrimination because of a person\'s transgender, intersex, or sexually indeterminate status is no less actionable than discrimination because of a person\'s identification with two religions, an unorthodox religion, or no religion at all. And "religious identity" can be just as fluid, variable, and difficult to define as "gender identity"; after all, both have "a deeply personal, internal genesis that lacks a fixed external referent." Sue Landsittel,\xa0Strange Bedfellows? Sex, Religion, and Transgender Identity Under Title VII,\xa0104 NW. U. L. REV. 1147, 1172 (2010) (advocating for "[t]he application of tests for religious identity to the problem of gender identity [because it] produces a more realistic, and therefore more appropriate, authentication framework than the current reliance on medical diagnoses and conformity with the gender binary").\n\n[5]\xa0On the other hand, there is also evidence that Stephens was fired only because of her nonconforming appearance and behavior at work, and not because of her transgender identity.\xa0See\xa0R. 53-6 (Rost Dep.',
    '[citation omitted]\n\n\xa0\n\n*1994 The program imposes no geographic limitation: Parents may direct tuition payments to schools inside or outside the State, or even in foreign countries. [citation omitted] In schools that qualify for the program because they are accredited, teachers need not be certified by the State,…and Maine’s curricular requirements do not apply…Single-sex schools are eligible. [citation omitted]\n\n\xa0\n\nPrior to 1981, parents could also direct the tuition assistance payments to religious schools. Indeed, in the 1979–1980 school year, over 200 Maine students opted to attend such schools through the tuition assistance program. App. 72. In 1981, however, Maine imposed a new requirement that any school receiving tuition assistance payments must be “a nonsectarian school in accordance with the First Amendment of the United States Constitution.” [citation omitted] That provision was enacted in response to an opinion by the Maine attorney general taking the position that public funding of private religious schools violated the Establishment Clause of the First Amendment. We subsequently held, however, that a benefit program under which private citizens “direct government aid to religious schools wholly as a result of their own genuine and independent private choice” does not offend the Establishment Clause. [citation omitted] Following our decision in Zelman, the Maine Legislature considered a proposed bill to repeal the “nonsectarian” requirement, but rejected it. App. 100, 108.\n\n\xa0\n\nThe “nonsectarian” requirement for participation in Maine’s tuition assistance program remains in effect today. The Department has stated that, in administering this requirement, it “considers a sectarian school to be one that is associated with a particular faith or belief system and which, in addition to teaching academic subjects, promotes the faith or belief system with which it is associated and/or presents the material taught through the lens of this faith.” [citation omitted] “The Department’s focus is on what the school teaches through its curriculum and related activities, and how the material is presented.” …“[A]ffiliation or association with a church or religious institution is one potential indicator of a sectarian school,” but “it is not dispositive.”\n\n\xa0\n\n\xa0\n\nB\n\nThis case concerns two families that live in SAUs that neither maintain their own secondary schools nor contract with any nearby secondary school. App. 70, 71. Petitioners David and Amy Carson reside in Glenburn, Maine. Id., at 74. When this litigation commenced, the Carsons’ daughter attended high school at Bangor Christian Schools (BCS), which was founded in 1970 as a ministry of Bangor Baptist Church. Id., at 74, 80. The Carsons sent their daughter to BCS because of the school’s high academic standards and because the school’s Christian worldview aligns with their sincerely held religious beliefs. Id., at 74. Given that BCS is a “sectarian” school that cannot qualify for tuition assistance payments under Maine’s program, id., at 80, the Carsons paid the tuition for their daughter to attend BCS themselves, id., at 74.\n\n\xa0\n\nPetitioners Troy and Angela Nelson live in Palermo, Maine. Id., at 78. When this litigation commenced, the Nelsons’ daughter attended high school at Erskine Academy, a secular private school, and their son attended middle school at Temple Academy, a “sectarian” school affiliated with *1995 Centerpoint Community Church. Id., at 78, 90, 91. The Nelsons sent their son to Temple Academy because they believed it offered him a high-quality education that aligned with their sincerely held religious beliefs. Id., at 78. While they wished to send their daughter to Temple Academy too, they could not afford to pay the cost of the Academy’s tuition for both of their children. Id., at 79.\n\n\xa0\n\nBCS and Temple Academy are both accredited by the New England Association of Schools and Colleges (NEASC), and the Department considers each school a “private school approved for attendance purposes” under the State’s compulsory attendance requirement. Id., at 80, 90. Yet because neither school qualifies as “nonsectarian,” neither is eligible to receive tuition payments under Maine’s tuition assistance program. Id., at 80, 90. Absent the “nonsectarian” requirement, the Carsons and the Nelsons would have asked their respective SAUs to pay the tuition to send their children to BCS and Temple Academy, respectively. Id., at 79.\n\n\xa0\n\nIn 2018, petitioners brought suit against the commissioner of the Maine Department of Education. Id., at 11–12.',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 768]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities.shape)
# [3, 3]

Evaluation

Metrics

Information Retrieval

• Datasets: dim_768, dim_512, dim_256, dim_128 and dim_64
• Evaluated with InformationRetrievalEvaluator

Metric	dim_768	dim_512	dim_256	dim_128	dim_64
cosine_accuracy@1	0.4839	0.4839	0.4516	0.4409	0.3978
cosine_accuracy@3	0.6989	0.7204	0.6882	0.6452	0.6022
cosine_accuracy@5	0.7957	0.7849	0.7957	0.7634	0.7097
cosine_accuracy@10	0.9247	0.9032	0.8817	0.8387	0.8065
cosine_precision@1	0.4839	0.4839	0.4516	0.4409	0.3978
cosine_precision@3	0.3799	0.3871	0.3656	0.3548	0.3405
cosine_precision@5	0.2839	0.286	0.2796	0.2731	0.2602
cosine_precision@10	0.172	0.1677	0.1656	0.1559	0.1538
cosine_recall@1	0.2177	0.2231	0.2088	0.1873	0.1586
cosine_recall@3	0.4884	0.5027	0.4718	0.4453	0.4059
cosine_recall@5	0.5883	0.5936	0.5806	0.5726	0.526
cosine_recall@10	0.7088	0.6944	0.6855	0.6541	0.6165
cosine_ndcg@10	0.5864	0.5845	0.565	0.5356	0.5019
cosine_mrr@10	0.5963	0.595	0.5674	0.5453	0.5082
cosine_map@100	0.4916	0.4987	0.4761	0.4511	0.4182

Training Details

Training Dataset

json

• Dataset: json
• Size: 842 training samples
• Columns: anchor and positive
• Approximate statistics based on the first 842 samples:

  	anchor	positive
  type	string	string
  details	min: 24 tokens mean: 42.46 tokens max: 68 tokens	min: 236 tokens mean: 962.01 tokens max: 1056 tokens

• Samples:

  anchor	positive
  Based on the court's ruling, under what circumstances can a college student be held accountable for off-campus speech, and how does this relate to the standards of professionalism in a professional school setting?	A serious question raised by Keefe in this case is whether the First Amendment protected his unprofessional speech from academic disadvantage because it was made in- on-line, off-campus Facebook postings. On appeal, Keefe framed this contention categorically, arguing that a college student may not be punished for off-campus speech unless it is speech that is unprotected by the First Amendment, such as obscenity. We reject this categorical contention. A student may demonstrate an unacceptable lack of professionalism off campus, as well as in the classroom, and by speech as well as conduct. See Yoder v. Univ. of Louisville, 526 Fed-Appx. 537, 545-46 (6th Cir.), cert. denied, — U.S. -, 134 S.Ct. 790, 187 L.Ed.2d 594 (2013); Tatro v. Univ. of Minn., 816 N.W.2d 509, 521 (Minn. 2012). Therefore, college administrators and educators in a professional school have discretion to require compliance with recognized standards of the profession, both on and off campus, “so long as their actions are ...
  Describe the two-step framework that Courts of Appeals have developed for analyzing Second Amendment challenges. What are the implications of the Supreme Court's decision to reject this framework in favor of a historical tradition-based approach?	Petitioners sued respondents for declaratory and injunctive relief under…42 U.S.C. § 1983, alleging that respondents violated their Second and Fourteenth Amendment rights by denying their unrestricted-license applications on the basis that they had failed to show “proper cause,” i.e., had failed to demonstrate a unique need for self-defense.   The District Court dismissed petitioners’ complaint and the Court of Appeals affirmed. [citation omitted] Both courts relied on [a] Court of Appeals’ prior decision…which had sustained New York’s proper-cause standard, holding that the requirement was “substantially related to the achievement of an important governmental interest.” [citation omitted]   We granted certiorari to decide whether New York’s denial of petitioners’ license applications violated the Constitution. [citation omitted]     II In Heller and McDonald, we held that the Second and Fourteenth Amendments protect an individual right to keep and bear arms for self-defense. ...
  Discuss the implications of the California Alien Land Law as it pertains to the rights of American citizens, specifically in the case of Fred Oyama. How does the law affect his privileges as a citizen, and what constitutional protections are being challenged?	269 Supreme Court of the United States OYAMA et al. v. STATE OF CALIFORNIA. No. 44.

• Loss: MatryoshkaLoss with these parameters:

  {
      "loss": "MultipleNegativesRankingLoss",
      "matryoshka_dims": [
          768,
          512,
          256,
          128,
          64
      ],
      "matryoshka_weights": [
          1,
          1,
          1,
          1,
          1
      ],
      "n_dims_per_step": -1
  }
  

Training Hyperparameters

Non-Default Hyperparameters

• eval_strategy: epoch
• per_device_train_batch_size: 16
• per_device_eval_batch_size: 16
• gradient_accumulation_steps: 32
• learning_rate: 2e-05
• num_train_epochs: 4
• lr_scheduler_type: cosine
• warmup_ratio: 0.1
• bf16: True
• tf32: True
• load_best_model_at_end: True
• optim: adamw_torch_fused
• batch_sampler: no_duplicates

All Hyperparameters

Click to expand

• overwrite_output_dir: False
• do_predict: False
• eval_strategy: epoch
• prediction_loss_only: True
• per_device_train_batch_size: 16
• per_device_eval_batch_size: 16
• per_gpu_train_batch_size: None
• per_gpu_eval_batch_size: None
• gradient_accumulation_steps: 32
• eval_accumulation_steps: None
• torch_empty_cache_steps: None
• learning_rate: 2e-05
• weight_decay: 0.0
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• max_grad_norm: 1.0
• num_train_epochs: 4
• max_steps: -1
• lr_scheduler_type: cosine
• lr_scheduler_kwargs: {}
• warmup_ratio: 0.1
• warmup_steps: 0
• log_level: passive
• log_level_replica: warning
• log_on_each_node: True
• logging_nan_inf_filter: True
• save_safetensors: True
• save_on_each_node: False
• save_only_model: False
• restore_callback_states_from_checkpoint: False
• no_cuda: False
• use_cpu: False
• use_mps_device: False
• seed: 42
• data_seed: None
• jit_mode_eval: False
• use_ipex: False
• bf16: True
• fp16: False
• fp16_opt_level: O1
• half_precision_backend: auto
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: True
• local_rank: 0
• ddp_backend: None
• tpu_num_cores: None
• tpu_metrics_debug: False
• debug: []
• dataloader_drop_last: False
• dataloader_num_workers: 0
• dataloader_prefetch_factor: None
• past_index: -1
• disable_tqdm: False
• remove_unused_columns: True
• label_names: None
• load_best_model_at_end: True
• ignore_data_skip: False
• fsdp: []
• fsdp_min_num_params: 0
• fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
• fsdp_transformer_layer_cls_to_wrap: None
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• deepspeed: None
• label_smoothing_factor: 0.0
• optim: adamw_torch_fused
• optim_args: None
• adafactor: False
• group_by_length: False
• length_column_name: length
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• skip_memory_metrics: True
• use_legacy_prediction_loop: False
• push_to_hub: False
• resume_from_checkpoint: None
• hub_model_id: None
• hub_strategy: every_save
• hub_private_repo: None
• hub_always_push: False
• gradient_checkpointing: False
• gradient_checkpointing_kwargs: None
• include_inputs_for_metrics: False
• include_for_metrics: []
• eval_do_concat_batches: True
• fp16_backend: auto
• push_to_hub_model_id: None
• push_to_hub_organization: None
• mp_parameters:
• auto_find_batch_size: False
• full_determinism: False
• torchdynamo: None
• ray_scope: last
• ddp_timeout: 1800
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• dispatch_batches: None
• split_batches: None
• include_tokens_per_second: False
• include_num_input_tokens_seen: False
• neftune_noise_alpha: None
• optim_target_modules: None
• batch_eval_metrics: False
• eval_on_start: False
• use_liger_kernel: False
• eval_use_gather_object: False
• average_tokens_across_devices: False
• prompts: None
• batch_sampler: no_duplicates
• multi_dataset_batch_sampler: proportional

Training Logs

Epoch	Step	dim_768_cosine_ndcg@10	dim_512_cosine_ndcg@10	dim_256_cosine_ndcg@10	dim_128_cosine_ndcg@10	dim_64_cosine_ndcg@10
0.6038	1	0.5604	0.5631	0.5303	0.4907	0.4335
1.6038	2	0.5836	0.5758	0.5715	0.5180	0.4846
2.6038	3	0.5768	0.5841	0.5652	0.5296	0.4940
3.6038	4	0.5864	0.5845	0.565	0.5356	0.5019

• The bold row denotes the saved checkpoint.

Framework Versions

• Python: 3.11.11
• Sentence Transformers: 3.4.1
• Transformers: 4.48.3
• PyTorch: 2.6.0+cu124
• Accelerate: 1.3.0
• Datasets: 3.3.0
• Tokenizers: 0.21.0

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

MatryoshkaLoss

@misc{kusupati2024matryoshka,
    title={Matryoshka Representation Learning},
    author={Aditya Kusupati and Gantavya Bhatt and Aniket Rege and Matthew Wallingford and Aditya Sinha and Vivek Ramanujan and William Howard-Snyder and Kaifeng Chen and Sham Kakade and Prateek Jain and Ali Farhadi},
    year={2024},
    eprint={2205.13147},
    archivePrefix={arXiv},
    primaryClass={cs.LG}
}

MultipleNegativesRankingLoss

@misc{henderson2017efficient,
    title={Efficient Natural Language Response Suggestion for Smart Reply},
    author={Matthew Henderson and Rami Al-Rfou and Brian Strope and Yun-hsuan Sung and Laszlo Lukacs and Ruiqi Guo and Sanjiv Kumar and Balint Miklos and Ray Kurzweil},
    year={2017},
    eprint={1705.00652},
    archivePrefix={arXiv},
    primaryClass={cs.CL}
}