Update README and config files - README.md

README.md

@@ -1,199 +1,307 @@
 ---
+language:
+- en
 library_name: transformers
-tags: []
+license: cc-by-4.0
+tags:
+- kl3m
+- legal
+- financial
+- mlm
+- roberta
+- embedding
+- uncased
+pipeline_tag: fill-mask
+tags_extended:
+- feature-extraction
+widget:
+ - text: "<|cls|> this credit agreement is made and entered into as of january 1, 2025, by and between acme corporation, a delaware <|mask|>, and first national bank, as lender. <|sep|>"
+ - example: "<|cls|> the<|mask|> agreement contains the entire understanding between <|sep|>"
+ - temperature: 0.7
+ - do_sample: true
+date: '2025-01-15T00:00:00.000Z'
 ---
 
-# Model Card for Model ID
-
-<!-- Provide a quick summary of what the model is/does. -->
-
+# kl3m-doc-micro-uncased-001
 
+`kl3m-doc-micro-uncased-001` is a domain-specific masked language model (MLM) based on the DeBERTa-v2 architecture, specifically designed for legal and financial document analysis. With approximately 118M parameters, it provides a specialized model for NLP tasks in both fill-mask prediction and feature extraction for document embeddings. This uncased variant is particularly useful for case-insensitive applications, maintaining strong performance while disregarding capitalization differences.
 
 ## Model Details
 
-### Model Description
-
-<!-- Provide a longer summary of what this model is. -->
-
-This is the model card of a 🤗 transformers model that has been pushed on the Hub. This model card has been automatically generated.
-
-- **Developed by:** [More Information Needed]
-- **Funded by [optional]:** [More Information Needed]
-- **Shared by [optional]:** [More Information Needed]
-- **Model type:** [More Information Needed]
-- **Language(s) (NLP):** [More Information Needed]
-- **License:** [More Information Needed]
-- **Finetuned from model [optional]:** [More Information Needed]
-
-### Model Sources [optional]
-
-<!-- Provide the basic links for the model. -->
-
-- **Repository:** [More Information Needed]
-- **Paper [optional]:** [More Information Needed]
-- **Demo [optional]:** [More Information Needed]
-
-## Uses
-
-<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
-
-### Direct Use
-
-<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
-
-[More Information Needed]
-
-### Downstream Use [optional]
-
-<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
-
-[More Information Needed]
-
-### Out-of-Scope Use
-
-<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
-
-[More Information Needed]
-
-## Bias, Risks, and Limitations
-
-<!-- This section is meant to convey both technical and sociotechnical limitations. -->
-
-[More Information Needed]
-
-### Recommendations
-
-<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
-
-Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
-
-## How to Get Started with the Model
-
-Use the code below to get started with the model.
-
-[More Information Needed]
-
-## Training Details
-
-### Training Data
-
-<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
-
-[More Information Needed]
-
-### Training Procedure
-
-<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
-
-#### Preprocessing [optional]
-
-[More Information Needed]
-
-
-#### Training Hyperparameters
-
-- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
-
-#### Speeds, Sizes, Times [optional]
-
-<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
-
-[More Information Needed]
-
-## Evaluation
-
-<!-- This section describes the evaluation protocols and provides the results. -->
-
-### Testing Data, Factors & Metrics
-
-#### Testing Data
-
-<!-- This should link to a Dataset Card if possible. -->
-
-[More Information Needed]
-
-#### Factors
-
-<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
-
-[More Information Needed]
-
-#### Metrics
-
-<!-- These are the evaluation metrics being used, ideally with a description of why. -->
-
-[More Information Needed]
-
-### Results
-
-[More Information Needed]
-
-#### Summary
-
-
-
-## Model Examination [optional]
-
-<!-- Relevant interpretability work for the model goes here -->
-
-[More Information Needed]
-
-## Environmental Impact
-
-<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
-
-Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
-
-- **Hardware Type:** [More Information Needed]
-- **Hours used:** [More Information Needed]
-- **Cloud Provider:** [More Information Needed]
-- **Compute Region:** [More Information Needed]
-- **Carbon Emitted:** [More Information Needed]
-
-## Technical Specifications [optional]
-
-### Model Architecture and Objective
-
-[More Information Needed]
-
-### Compute Infrastructure
-
-[More Information Needed]
-
-#### Hardware
-
-[More Information Needed]
-
-#### Software
-
-[More Information Needed]
-
-## Citation [optional]
-
-<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
-
-**BibTeX:**
-
-[More Information Needed]
-
-**APA:**
+- **Architecture**: DeBERTa-v2
+- **Size**: 118M parameters
+- **Hidden Size**: 512
+- **Layers**: 16
+- **Attention Heads**: 16
+- **Intermediate Size**: 2048
+- **Max Position Embeddings**: 512
+- **Tokenizer**: [alea-institute/kl3m-004-128k-uncased](https://huggingface.co/alea-institute/kl3m-004-128k-uncased)
+- **Vector Dimension**: 512 (hidden_size)
+- **Pooling Strategy**: CLS token or mean pooling
 
-[More Information Needed]
+## Use Cases
 
-## Glossary [optional]
+This model is particularly useful for:
 
-<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
+- Lightweight document classification in legal and financial domains
+- Entity recognition for specialized terminology
+- Understanding legal citations and references
+- Filling in missing terms in legal documents
+- Feature extraction for downstream legal analysis tasks
+- Document similarity and retrieval tasks where capitalization is not significant
+- Edge computing applications with limited resources
 
-[More Information Needed]
+The uncased nature of this model makes it more efficient for scenarios where case distinctions are not important to the task, while its micro size makes it suitable for severely resource-constrained environments.
 
-## More Information [optional]
+## Standard Test Examples
+
+Using our standardized test examples for comparing embedding models:
 
-[More Information Needed]
+### Fill-Mask Results
+
+1. **Contract Clause Heading**:  
+   `"<|cls|> 8. representations and<|mask|>. each party hereby represents and warrants to the other party as of the date hereof as follows: <|sep|>"`
+
+   Top 5 predictions:
+   1. warranties (0.8665)
+   2. warrants (0.0857)
+   3. warrant (0.0108)
+   4. covenants (0.0079)
+   5. agreements (0.0057)
+
+2. **Defined Term Example**:  
+   `"<|cls|> \"effective<|mask|>\" means the date on which all conditions precedent set forth in article v are satisfied or waived by the administrative agent. <|sep|>"`
+
+   Top 5 predictions:
+   1. date (0.9979)
+   2. time (0.0011)
+   3. day (0.0004)
+   4. dates (0.0002)
+   5. deadline (0.0001)
+
+3. **Regulation Example**:  
+   `"<|cls|> all transactions shall comply with the requirements set forth in the truth in<|mask|> act and its implementing regulation z. <|sep|>"`
+
+   Top 5 predictions:
+   1. the (0.4650)
+   2. this (0.4385)
+   3. such (0.0104)
+   4. any (0.0050)
+   5. an (0.0045)
+
+### Document Similarity Results
+
+Using the standardized document examples for embeddings:
+
+| Document Pair | Cosine Similarity (CLS token) | Cosine Similarity (Mean pooling) |
+|---------------|-------------------------------|----------------------------------|
+| Court Complaint vs. Consumer Terms | 0.607 | 0.723 |
+| Court Complaint vs. Credit Agreement | 0.631 | 0.837 |
+| Consumer Terms vs. Credit Agreement | 0.744 | 0.762 |
+
+The micro-sized model shows strong document similarity performance despite its compact size, with mean pooling showing particularly good results for capturing similarity between related legal documents. The highest similarity with CLS tokens is between Consumer Terms and Credit Agreement (0.744), while with mean pooling, the highest similarity is between Court Complaint and Credit Agreement (0.837).
+
+## Usage
+
+### Masked Language Modeling
+
+For masked language modeling tasks, you can use the simple pipeline API:
+
+```python
+from transformers import pipeline
+
+# Load the fill-mask pipeline with the model
+fill_mask = pipeline('fill-mask', model="alea-institute/kl3m-doc-micro-uncased-001")
+
+# Example: Contract clause heading
+# Note the mask token placement - directly adjacent to "and" without space
+text = "<|cls|> 8. representations and<|mask|>. each party hereby represents and warrants to the other party as of the date hereof as follows: <|sep|>"
+results = fill_mask(text)
+
+# Display predictions
+print("Top predictions:")
+for result in results:
+    print(f"- {result['token_str']} (score: {result['score']:.4f})")
+
+# Output:
+# Top predictions:
+# - warranties (score: 0.8665)
+# - warrants (score: 0.0857)
+# - warrant (score: 0.0108)
+# - covenants (score: 0.0079)
+# - agreements (score: 0.0057)
+```
+
+### Feature Extraction for Embeddings
+
+For document embeddings and similarity calculations, you can also use the pipeline API:
+
+```python
+from transformers import pipeline
+import numpy as np
+from sklearn.metrics.pairwise import cosine_similarity
+
+# Load the feature-extraction pipeline
+extractor = pipeline('feature-extraction', model="alea-institute/kl3m-doc-micro-uncased-001", return_tensors=True)
+
+# Example legal documents
+texts = [
+    # Court Complaint
+    "<|cls|> in the united states district court for the eastern district of pennsylvania\n\njohn doe,\nplaintiff,\n\nvs.\n\nacme corporation,\ndefendant. <|sep|>",
+    
+    # Consumer Terms
+    "<|cls|> terms and conditions\n\nlast updated: april 10, 2025\n\nthese terms and conditions govern your access to and use of the service. <|sep|>",
+    
+    # Credit Agreement
+    "<|cls|> credit agreement\n\ndated as of april 10, 2025\n\namong\n\nacme borrower inc.,\nas the borrower,\n\nand bank of finance,\nas administrative agent. <|sep|>"
+]
+
+# Strategy 1: CLS token embeddings
+cls_embeddings = []
+for text in texts:
+    features = extractor(text)
+    # Get the CLS token (first token) embedding
+    features_array = features[0].numpy() if hasattr(features[0], 'numpy') else features[0]
+    cls_embedding = features_array[0]
+    cls_embeddings.append(cls_embedding)
+
+# Calculate similarity between documents using CLS tokens
+cls_similarity = cosine_similarity(np.vstack(cls_embeddings))
+print("\nDocument similarity (CLS token):")
+print(np.round(cls_similarity, 3))
+# Output:
+# [[1.    0.607 0.631]
+#  [0.607 1.    0.744]
+#  [0.631 0.744 1.   ]]
+
+# Strategy 2: Mean pooling
+mean_embeddings = []
+for text in texts:
+    features = extractor(text)
+    # Average over all tokens
+    features_array = features[0].numpy() if hasattr(features[0], 'numpy') else features[0]
+    mean_embedding = np.mean(features_array, axis=0)
+    mean_embeddings.append(mean_embedding)
+
+# Calculate similarity using mean pooling
+mean_similarity = cosine_similarity(np.vstack(mean_embeddings))
+print("\nDocument similarity (Mean pooling):")
+print(np.round(mean_similarity, 3))
+# Output:
+# [[1.    0.723 0.837]
+#  [0.723 1.    0.762]
+#  [0.837 0.762 1.   ]]
+
+# Print pairwise similarities
+doc_names = ["Court Complaint", "Consumer Terms", "Credit Agreement"]
+print("\nPairwise similarities:")
+for i in range(len(doc_names)):
+    for j in range(i+1, len(doc_names)):
+        print(f"{doc_names[i]} vs. {doc_names[j]}:")
+        print(f"  - CLS token: {cls_similarity[i, j]:.4f}")
+        print(f"  - Mean pooling: {mean_similarity[i, j]:.4f}")
+# Output:
+# Pairwise similarities:
+# Court Complaint vs. Consumer Terms:
+#   - CLS token: 0.6073
+#   - Mean pooling: 0.7233
+# Court Complaint vs. Credit Agreement:
+#   - CLS token: 0.6314
+#   - Mean pooling: 0.8370
+# Consumer Terms vs. Credit Agreement:
+#   - CLS token: 0.7435
+#   - Mean pooling: 0.7620
+```
+
+## Training
+
+The model was trained on a diverse corpus of legal and financial documents, ensuring high-quality performance in these domains. It leverages the KL3M tokenizer which provides 9-17% more efficient tokenization for domain-specific content than cl100k_base or the LLaMA/Mistral tokenizers.
+
+Training included both masked language modeling (MLM) objectives and attention to dense document representation for retrieval and classification tasks. The model was trained on lowercase text to improve efficiency and reduce model size.
+
+## Intended Usage
+
+This model is intended for both:
+
+1. **Masked Language Modeling**: Filling in missing words/terms in legal and financial documents
+2. **Document Embedding**: Generating fixed-length vector representations for document similarity and classification
+
+It is particularly well-suited for resource-constrained environments and applications where case-sensitivity is not important.
+
+## Special Tokens
+
+This model includes the following special tokens:
+
+- CLS token: `<|cls|>` (ID: 5) - Used for the beginning of input text
+- MASK token: `<|mask|>` (ID: 6) - Used to mark tokens for prediction
+- SEP token: `<|sep|>` (ID: 4) - Used for the end of input text
+- PAD token: `<|pad|>` (ID: 2) - Used for padding sequences to a uniform length
+- BOS token: `<|start|>` (ID: 0) - Beginning of sequence
+- EOS token: `<|end|>` (ID: 1) - End of sequence
+- UNK token: `<|unk|>` (ID: 3) - Unknown token
+
+Important usage notes:
+
+When using the MASK token for predictions, be aware that this model uses a **space-prefixed BPE tokenizer**. The <|mask|> token should be placed IMMEDIATELY after the previous token with NO space, because most tokens in this tokenizer have an initial space encoded within them. For example: `"word<|mask|>"` rather than `"word <|mask|>"`.
+   
+This space-aware placement is crucial for getting accurate predictions.
+
+## Limitations
+
+While extremely compact, this model has significant limitations:
+
+- Limited parameter count (118M) means it captures less nuance than larger language models
+- Uncased nature means it cannot distinguish between different capitalizations
+- Primarily focused on English legal and financial texts
+- Best suited for domain-specific rather than general-purpose tasks
+- Maximum sequence length of 512 tokens may require chunking for lengthy documents
+- Requires domain expertise to interpret results effectively
+
+## References
+
+- [KL3M Tokenizers: A Family of Domain-Specific and Character-Level Tokenizers for Legal, Financial, and Preprocessing Applications](https://arxiv.org/abs/2503.17247)
+- [The KL3M Data Project: Copyright-Clean Training Resources for Large Language Models](https://arxiv.org/abs/2504.07854)
+
+## Citation
+
+If you use this model in your research, please cite:
+
+```bibtex
+@misc{kl3m-doc-micro-uncased-001,
+  author = {ALEA Institute},
+  title = {kl3m-doc-micro-uncased-001: A Domain-Specific Uncased Language Model for Legal and Financial Text Analysis},
+  year = {2025},
+  publisher = {Hugging Face},
+  howpublished = {\url{https://huggingface.co/alea-institute/kl3m-doc-micro-uncased-001}}
+}
+
+@article{bommarito2025kl3m,
+  title={KL3M Tokenizers: A Family of Domain-Specific and Character-Level Tokenizers for Legal, Financial, and Preprocessing Applications},
+  author={Bommarito, Michael J and Katz, Daniel Martin and Bommarito, Jillian},
+  journal={arXiv preprint arXiv:2503.17247},
+  year={2025}
+}
+
+@misc{bommarito2025kl3mdata,
+  title={The KL3M Data Project: Copyright-Clean Training Resources for Large Language Models},
+  author={Bommarito II, Michael J. and Bommarito, Jillian and Katz, Daniel Martin},
+  year={2025},
+  eprint={2504.07854},
+  archivePrefix={arXiv},
+  primaryClass={cs.CL}
+}
+```
 
-## Model Card Authors [optional]
+## License
+
+This model is licensed under [CC-BY 4.0](https://creativecommons.org/licenses/by/4.0/).
 
-[More Information Needed]
+## Contact
 
-## Model Card Contact
+The KL3M model family is maintained by the [ALEA Institute](https://aleainstitute.ai). For technical support, collaboration opportunities, or general inquiries:
+ 
+- Email: [email protected]
+- Website: https://aleainstitute.ai
+- GitHub: https://github.com/alea-institute/kl3m-model-research
 
-[More Information Needed]
+![logo](https://aleainstitute.ai/images/alea-logo-ascii-1x1.png)