Create README.md

README.md

@@ -0,0 +1,83 @@
+---
+base_model:
+- nvidia/Llama-3_3-Nemotron-Super-49B-v1
+tags:
+- L-Mul,
+- optimazation
+- quantization
+- text-generation
+- research
+- experimental
+license: other
+---
+
+# Model Card for nvidia/Llama-3_3-Nemotron-Super-49B-v1-LMUL
+
+This model is a derivative of `nvidia/Llama-3_3-Nemotron-Super-49B-v1`, modified to use a custom attention mechanism defined by the `l_mul_attention` function from the `lmul` library.
+
+## Model Details
+
+- **Original Model:** [nvidia/Llama-3_3-Nemotron-Super-49B-v1](https://huggingface.co/nvidia/Llama-3_3-Nemotron-Super-49B-v1)
+- **Architecture:** `DeciLM` (`decilm`)
+- **Modification:** The `forward` method of the `DeciAttention` module has been replaced (monkey-patched) with a custom implementation that utilizes the `l_mul_attention` logic. Note that in some blocks of the original model, the attention layer is skipped entirely; those blocks are unaffected by this modification.
+
+## Scientific Rationale
+
+This model was modified as part of a research project investigating alternative attention mechanisms in large language models. The `l_mul_attention` function implements a novel approach to calculating attention scores, and this model serves as a test case for evaluating its performance, efficiency, and impact on reasoning and generation tasks compared to the standard attention implementation.
+
+By releasing this model, we hope to encourage further research into non-standard attention mechanisms and provide a practical example for the community to build upon.
+
+## How to Get Started
+
+You can use this model with the standard `transformers` library pipeline. Because the base model uses a custom architecture, you must use `trust_remote_code=True` when loading it.
+
+```python
+from transformers import AutoTokenizer, AutoModelForCausalLM
+import torch
+
+# Make sure to log in with your Hugging Face token if the model is private
+# from huggingface_hub import login
+# login("your-hf-token")
+
+model_id = "YOUR_HF_USERNAME/Llama-3_3-Nemotron-Super-49B-v1-LMUL" # Replace with your HF username
+device = "cuda" if torch.cuda.is_available() else "cpu"
+
+tokenizer = AutoTokenizer.from_pretrained(model_id)
+model = AutoModelForCausalLM.from_pretrained(
+    model_id,
+    torch_dtype=torch.bfloat16,
+    device_map="auto",
+    trust_remote_code=True  # Important! Required by the base model
+)
+
+# The base model uses a system prompt to control reasoning
+thinking = "on" # or "off"
+messages = [
+    {"role": "system", "content": f"detailed thinking {thinking}"},
+    {"role": "user", "content": "What is the airspeed velocity of an unladen swallow?"}
+]
+
+# Note: The original model's tokenizer does not have a chat template.
+# You must apply it manually or use the pipeline as shown in the original model card.
+# For simplicity, we'll format the prompt manually here.
+prompt = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
+model_inputs = tokenizer([prompt], return_tensors="pt").to(model.device)
+
+generated_ids = model.generate(
+    **model_inputs,
+    max_new_tokens=512,
+    temperature=0.6,
+    top_p=0.95
+)
+response = tokenizer.batch_decode(generated_ids, skip_special_tokens=True)[0]
+
+print(response)
+```
+
+## Intended Uses & Limitations
+
+This model is intended primarily for research purposes. Its performance on standard benchmarks has not been fully evaluated. The custom attention mechanism may introduce unexpected behaviors or limitations not present in the original model. The original model has specific prompting requirements (e.g., for controlling reasoning) which should be followed.
+
+## Licensing Information
+
+This model is released under the `nvidia-open-model-license`, which is the same license as the base model, `nvidia/Llama-3_3-Nemotron-Super-49B-v1`. By using this model, you agree to the terms of the original license. It is your responsibility to ensure compliance with all applicable licenses and regulations. The model is also built upon Meta Llama 3, and its use is subject to the Llama 3.3 Community License Agreement.