kl3m-doc-micro-uncased-001

kl3m-doc-micro-uncased-001 is a domain-specific masked language model (MLM) based on the DeBERTa-v2 architecture, specifically designed for legal and financial document analysis. With approximately 118M parameters, it provides a specialized model for NLP tasks in both fill-mask prediction and feature extraction for document embeddings. This uncased variant is particularly useful for case-insensitive applications, maintaining strong performance while disregarding capitalization differences.

Model Details

Architecture: DeBERTa-v2
Size: 118M parameters
Hidden Size: 512
Layers: 16
Attention Heads: 16
Intermediate Size: 2048
Max Position Embeddings: 512
Tokenizer: alea-institute/kl3m-004-128k-uncased
Vector Dimension: 512 (hidden_size)
Pooling Strategy: CLS token or mean pooling

Use Cases

This model is particularly useful for:

Lightweight document classification in legal and financial domains
Entity recognition for specialized terminology
Understanding legal citations and references
Filling in missing terms in legal documents
Feature extraction for downstream legal analysis tasks
Document similarity and retrieval tasks where capitalization is not significant
Edge computing applications with limited resources

The uncased nature of this model makes it more efficient for scenarios where case distinctions are not important to the task, while its micro size makes it suitable for severely resource-constrained environments.

Standard Test Examples

Using our standardized test examples for comparing embedding models:

Fill-Mask Results

Contract Clause Heading:
"<|cls|> 8. representations and<|mask|>. each party hereby represents and warrants to the other party as of the date hereof as follows: <|sep|>"

Top 5 predictions:
1. warranties (0.8665)
2. warrants (0.0857)
3. warrant (0.0108)
4. covenants (0.0079)
5. agreements (0.0057)
Defined Term Example:
"<|cls|> \"effective<|mask|>\" means the date on which all conditions precedent set forth in article v are satisfied or waived by the administrative agent. <|sep|>"

Top 5 predictions:
1. date (0.9979)
2. time (0.0011)
3. day (0.0004)
4. dates (0.0002)
5. deadline (0.0001)
Regulation Example:
"<|cls|> all transactions shall comply with the requirements set forth in the truth in<|mask|> act and its implementing regulation z. <|sep|>"

Top 5 predictions:
1. the (0.4650)
2. this (0.4385)
3. such (0.0104)
4. any (0.0050)
5. an (0.0045)

Document Similarity Results

Using the standardized document examples for embeddings:

Document Pair	Cosine Similarity (CLS token)	Cosine Similarity (Mean pooling)
Court Complaint vs. Consumer Terms	0.607	0.723
Court Complaint vs. Credit Agreement	0.631	0.837
Consumer Terms vs. Credit Agreement	0.744	0.762

The micro-sized model shows strong document similarity performance despite its compact size, with mean pooling showing particularly good results for capturing similarity between related legal documents. The highest similarity with CLS tokens is between Consumer Terms and Credit Agreement (0.744), while with mean pooling, the highest similarity is between Court Complaint and Credit Agreement (0.837).

Usage

Masked Language Modeling

For masked language modeling tasks, you can use the simple pipeline API:

from transformers import pipeline

# Load the fill-mask pipeline with the model
fill_mask = pipeline('fill-mask', model="alea-institute/kl3m-doc-micro-uncased-001")

# Example: Contract clause heading
# Note the mask token placement - directly adjacent to "and" without space
text = "<|cls|> 8. representations and<|mask|>. each party hereby represents and warrants to the other party as of the date hereof as follows: <|sep|>"
results = fill_mask(text)

# Display predictions
print("Top predictions:")
for result in results:
    print(f"- {result['token_str']} (score: {result['score']:.4f})")

# Output:
# Top predictions:
# - warranties (score: 0.8665)
# - warrants (score: 0.0857)
# - warrant (score: 0.0108)
# - covenants (score: 0.0079)
# - agreements (score: 0.0057)

Feature Extraction for Embeddings

For document embeddings and similarity calculations, you can also use the pipeline API:

from transformers import pipeline
import numpy as np
from sklearn.metrics.pairwise import cosine_similarity

# Load the feature-extraction pipeline
extractor = pipeline('feature-extraction', model="alea-institute/kl3m-doc-micro-uncased-001", return_tensors=True)

# Example legal documents
texts = [
    # Court Complaint
    "<|cls|> in the united states district court for the eastern district of pennsylvania\n\njohn doe,\nplaintiff,\n\nvs.\n\nacme corporation,\ndefendant. <|sep|>",
    
    # Consumer Terms
    "<|cls|> terms and conditions\n\nlast updated: april 10, 2025\n\nthese terms and conditions govern your access to and use of the service. <|sep|>",
    
    # Credit Agreement
    "<|cls|> credit agreement\n\ndated as of april 10, 2025\n\namong\n\nacme borrower inc.,\nas the borrower,\n\nand bank of finance,\nas administrative agent. <|sep|>"
]

# Strategy 1: CLS token embeddings
cls_embeddings = []
for text in texts:
    features = extractor(text)
    # Get the CLS token (first token) embedding
    features_array = features[0].numpy() if hasattr(features[0], 'numpy') else features[0]
    cls_embedding = features_array[0]
    cls_embeddings.append(cls_embedding)

# Calculate similarity between documents using CLS tokens
cls_similarity = cosine_similarity(np.vstack(cls_embeddings))
print("\nDocument similarity (CLS token):")
print(np.round(cls_similarity, 3))
# Output:
# [[1.    0.607 0.631]
#  [0.607 1.    0.744]
#  [0.631 0.744 1.   ]]

# Strategy 2: Mean pooling
mean_embeddings = []
for text in texts:
    features = extractor(text)
    # Average over all tokens
    features_array = features[0].numpy() if hasattr(features[0], 'numpy') else features[0]
    mean_embedding = np.mean(features_array, axis=0)
    mean_embeddings.append(mean_embedding)

# Calculate similarity using mean pooling
mean_similarity = cosine_similarity(np.vstack(mean_embeddings))
print("\nDocument similarity (Mean pooling):")
print(np.round(mean_similarity, 3))
# Output:
# [[1.    0.723 0.837]
#  [0.723 1.    0.762]
#  [0.837 0.762 1.   ]]

# Print pairwise similarities
doc_names = ["Court Complaint", "Consumer Terms", "Credit Agreement"]
print("\nPairwise similarities:")
for i in range(len(doc_names)):
    for j in range(i+1, len(doc_names)):
        print(f"{doc_names[i]} vs. {doc_names[j]}:")
        print(f"  - CLS token: {cls_similarity[i, j]:.4f}")
        print(f"  - Mean pooling: {mean_similarity[i, j]:.4f}")
# Output:
# Pairwise similarities:
# Court Complaint vs. Consumer Terms:
#   - CLS token: 0.6073
#   - Mean pooling: 0.7233
# Court Complaint vs. Credit Agreement:
#   - CLS token: 0.6314
#   - Mean pooling: 0.8370
# Consumer Terms vs. Credit Agreement:
#   - CLS token: 0.7435
#   - Mean pooling: 0.7620

Training

The model was trained on a diverse corpus of legal and financial documents, ensuring high-quality performance in these domains. It leverages the KL3M tokenizer which provides 9-17% more efficient tokenization for domain-specific content than cl100k_base or the LLaMA/Mistral tokenizers.

Training included both masked language modeling (MLM) objectives and attention to dense document representation for retrieval and classification tasks. The model was trained on lowercase text to improve efficiency and reduce model size.

Intended Usage

This model is intended for both:

Masked Language Modeling: Filling in missing words/terms in legal and financial documents
Document Embedding: Generating fixed-length vector representations for document similarity and classification

It is particularly well-suited for resource-constrained environments and applications where case-sensitivity is not important.

Special Tokens

This model includes the following special tokens:

CLS token: <|cls|> (ID: 5) - Used for the beginning of input text
MASK token: <|mask|> (ID: 6) - Used to mark tokens for prediction
SEP token: <|sep|> (ID: 4) - Used for the end of input text
PAD token: <|pad|> (ID: 2) - Used for padding sequences to a uniform length
BOS token: <|start|> (ID: 0) - Beginning of sequence
EOS token: <|end|> (ID: 1) - End of sequence
UNK token: <|unk|> (ID: 3) - Unknown token

Important usage notes:

When using the MASK token for predictions, be aware that this model uses a space-prefixed BPE tokenizer. The <|mask|> token should be placed IMMEDIATELY after the previous token with NO space, because most tokens in this tokenizer have an initial space encoded within them. For example: "word<|mask|>" rather than "word <|mask|>".

This space-aware placement is crucial for getting accurate predictions.

Limitations

While extremely compact, this model has significant limitations:

Limited parameter count (118M) means it captures less nuance than larger language models
Uncased nature means it cannot distinguish between different capitalizations
Primarily focused on English legal and financial texts
Best suited for domain-specific rather than general-purpose tasks
Maximum sequence length of 512 tokens may require chunking for lengthy documents
Requires domain expertise to interpret results effectively

References

Citation

If you use this model in your research, please cite:

@misc{kl3m-doc-micro-uncased-001,
  author = {ALEA Institute},
  title = {kl3m-doc-micro-uncased-001: A Domain-Specific Uncased Language Model for Legal and Financial Text Analysis},
  year = {2025},
  publisher = {Hugging Face},
  howpublished = {\url{https://huggingface.co/alea-institute/kl3m-doc-micro-uncased-001}}
}

@article{bommarito2025kl3m,
  title={KL3M Tokenizers: A Family of Domain-Specific and Character-Level Tokenizers for Legal, Financial, and Preprocessing Applications},
  author={Bommarito, Michael J and Katz, Daniel Martin and Bommarito, Jillian},
  journal={arXiv preprint arXiv:2503.17247},
  year={2025}
}

@misc{bommarito2025kl3mdata,
  title={The KL3M Data Project: Copyright-Clean Training Resources for Large Language Models},
  author={Bommarito II, Michael J. and Bommarito, Jillian and Katz, Daniel Martin},
  year={2025},
  eprint={2504.07854},
  archivePrefix={arXiv},
  primaryClass={cs.CL}
}

License

This model is licensed under CC-BY 4.0.

Contact

The KL3M model family is maintained by the ALEA Institute. For technical support, collaboration opportunities, or general inquiries:

Email: [email protected]
Website: https://aleainstitute.ai
GitHub: https://github.com/alea-institute/kl3m-model-research

alea-institute
/

kl3m-doc-micro-uncased-001