Upload folder using huggingface_hub

README.md

@@ -0,0 +1,161 @@
+---
+language: tr
+license: apache-2.0
+library_name: transformers
+tags:
+- text-generation
+- turkish
+- deepseek
+- moe
+- mla
+- pytorch
+- causal-lm
+datasets:
+- tr_wikipedia
+widget:
+- text: "Merhaba dünya"
+  example_title: "Turkish Greeting"
+- text: "Türkiye'nin başkenti"
+  example_title: "Turkish Geography"
+- text: "Yapay zeka"
+  example_title: "Turkish Technology"
+---
+
+# Turkish DeepSeek Model
+
+Bu model, DeepSeek mimarisi kullanılarak Türkçe metinler üzerinde eğitilmiş bir dil modelidir. Multi-head Latent Attention (MLA) ve Mixture of Experts (MoE) teknolojilerini içerir.
+
+## Model Özellikleri
+
+- **Parametre Sayısı**: ~192M
+- **Kelime Hazinesi**: 50,256 token
+- **Bağlam Uzunluğu**: 256 token
+- **Dil**: Türkçe (tr)
+- **Mimarisi**: DeepSeek with MLA + MoE
+
+## Teknik Detaylar
+
+- **Gizli Boyut**: 1024
+- **Katman Sayısı**: 6 (1 yoğun + 5 MoE)
+- **Attention Head**: 8
+- **MoE Uzmanları**: 4 yönlendirilmiş + 2 paylaşımlı
+- **Aktif Uzman**: 2 per token
+
+## Kullanım
+
+### Temel Kullanım
+
+```python
+import torch
+from transformers import AutoTokenizer, AutoModelForCausalLM
+
+# Model ve tokenizer'ı yükle
+model = AutoModelForCausalLM.from_pretrained("your-username/turkish-deepseek", trust_remote_code=True)
+tokenizer = AutoTokenizer.from_pretrained("your-username/turkish-deepseek")
+
+# Metin üretimi
+prompt = "Merhaba dünya"
+inputs = tokenizer(prompt, return_tensors="pt")
+with torch.no_grad():
+    outputs = model.generate(
+        **inputs,
+        max_length=50,
+        temperature=0.7,
+        do_sample=True,
+        pad_token_id=tokenizer.pad_token_id
+    )
+
+generated_text = tokenizer.decode(outputs[0], skip_special_tokens=True)
+print(generated_text)
+```
+
+### Orijinal Implementation ile Kullanım
+
+```python
+# Orijinal implementasyonu kullanmak için
+import torch
+import sentencepiece as spm
+
+# Tokenizer'ı yükle
+tokenizer = spm.SentencePieceProcessor()
+tokenizer.load("tokenizer.model")
+
+# Model checkpoint'ini yükle
+checkpoint = torch.load("pytorch_model.bin", map_location="cpu")
+
+# Orijinal model sınıfınızı kullanarak yükleyin
+# from your_original_implementation import Transformer, ModelArgs
+# model = Transformer(args)
+# model.load_state_dict(checkpoint)
+```
+
+## Eğitim Verisi
+
+- **Kaynak**: Türkçe Wikipedia
+- **Tokenization**: SentencePiece BPE
+- **Kelime Hazinesi**: Türkçe diline optimize edilmiş
+
+## Model Mimarisi
+
+### Multi-head Latent Attention (MLA)
+- Sıkıştırılmış key-value temsilleri (rank 256)
+- Ayrı no-position ve position encoding bileşenleri
+- Uzun diziler için verimli bellek kullanımı
+
+### Mixture of Experts (MoE)
+- Top-2 yönlendirme ve yük dengeleme
+- Ortak desenler için paylaşımlı uzmanlar
+- Seyrek aktivasyon ile azaltılmış hesaplama
+
+### RoPE with YaRN Scaling
+- Frekans ölçekleme ile rotational position embedding
+- Eğitim uzunluğunun ötesinde genişletilmiş bağlam desteği
+- Temel frekans: 10000.0
+
+## Performans
+
+- **Çıkarım**: Türkçe metin üretimi için optimize edilmiş
+- **Bellek**: MLA, KV cache boyutunu azaltır
+- **Hız**: MoE, kontrollü hesaplama ile daha büyük kapasiteye olanak tanır
+
+## Sınırlamalar
+
+- Ağırlıklı olarak Türkçe Wikipedia üzerinde eğitilmiş (sınırlı alan kapsamı)
+- Bağlam uzunluğu 256 token ile sınırlı
+- Eğitim verisinde mevcut önyargılar sergileyebilir
+
+## Alıntı
+
+Bu modeli kullanırsanız, lütfen alıntı yapın:
+
+```bibtex
+@misc{turkish-deepseek,
+  title={Turkish DeepSeek Language Model},
+  author={Your Name},
+  year={2024},
+  url={https://huggingface.co/your-username/turkish-deepseek}
+}
+```
+
+## Lisans
+
+Apache 2.0 License
+
+## Model Card Authors
+
+[Your Name]
+
+---
+
+## English Summary
+
+This is a Turkish language model based on the DeepSeek architecture, featuring Multi-head Latent Attention (MLA) and Mixture of Experts (MoE). The model has ~192M parameters and was trained on Turkish Wikipedia data.
+
+### Key Features
+- **Architecture**: DeepSeek with advanced MLA and MoE components
+- **Language**: Turkish (tr)
+- **Training**: Turkish Wikipedia corpus
+- **Vocabulary**: 50,256 tokens optimized for Turkish
+
+### Usage
+Load with `trust_remote_code=True` to use the custom implementation, or use the provided model files directly.

__init__.py

@@ -0,0 +1,31 @@
+# flake8: noqa
+# There's no way to ignore "F401 '...' imported but unused" warnings in this
+# module, but to preserve other warnings. So, don't check this module at all.
+
+# Copyright 2024 The HuggingFace Team. All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# Simple standalone imports for DeepSeek model
+from configuration_deepseek import (DEEPSEEK_PRETRAINED_CONFIG_ARCHIVE_MAP,
+                                    DeepSeekConfig)
+from modeling_deepseek import (DeepSeekForCausalLM, DeepSeekModel,
+                               DeepSeekPreTrainedModel)
+
+__all__ = [
+    "DEEPSEEK_PRETRAINED_CONFIG_ARCHIVE_MAP",
+    "DeepSeekConfig", 
+    "DeepSeekForCausalLM",
+    "DeepSeekModel",
+    "DeepSeekPreTrainedModel",
+] 

config.json

@@ -0,0 +1,41 @@
+{
+  "architectures": [
+    "TurkishDeepSeek"
+  ],
+  "model_type": "turkish_deepseek",
+  "vocab_size": 50256,
+  "hidden_size": 1024,
+  "num_hidden_layers": 6,
+  "num_dense_layers": 1,
+  "num_attention_heads": 8,
+  "intermediate_size": 4096,
+  "moe_intermediate_size": 704,
+  "num_routed_experts": 4,
+  "num_shared_experts": 2,
+  "num_activated_experts": 2,
+  "max_position_embeddings": 256,
+  "q_lora_rank": 0,
+  "kv_lora_rank": 256,
+  "qk_nope_head_dim": 64,
+  "qk_rope_head_dim": 32,
+  "v_head_dim": 64,
+  "original_seq_len": 512,
+  "rope_theta": 10000.0,
+  "rope_factor": 40,
+  "beta_fast": 32,
+  "beta_slow": 1,
+  "mscale": 1.0,
+  "rms_norm_eps": 0.001,
+  "initializer_range": 0.02,
+  "use_cache": true,
+  "pad_token_id": 0,
+  "bos_token_id": 2,
+  "eos_token_id": 3,
+  "tie_word_embeddings": false,
+  "torch_dtype": "float32",
+  "auto_map": {
+    "AutoConfig": "configuration_deepseek.TurkishDeepSeekConfig",
+    "AutoModel": "modeling_deepseek.TurkishDeepSeekModel",
+    "AutoModelForCausalLM": "modeling_deepseek.TurkishDeepSeekForCausalLM"
+  }
+}

configuration_deepseek.py

@@ -0,0 +1,188 @@
+"""
+DeepSeek model configuration
+"""
+
+from transformers.configuration_utils import PretrainedConfig
+from transformers.utils import logging
+
+logger = logging.get_logger(__name__)
+
+DEEPSEEK_PRETRAINED_CONFIG_ARCHIVE_MAP = {}
+
+
+class DeepSeekConfig(PretrainedConfig):
+    r"""
+    This is the configuration class to store the configuration of a [`DeepSeekModel`]. It is used to instantiate a
+    DeepSeek model according to the specified arguments, defining the model architecture. Instantiating a configuration
+    with the defaults will yield a similar configuration to that of the DeepSeek-V3
+    [deepseek-ai/DeepSeek-V3](https://huggingface.co/deepseek-ai/DeepSeek-V3) architecture.
+
+    Configuration objects inherit from [`PretrainedConfig`] and can be used to control the model outputs. Read the
+    documentation from [`PretrainedConfig`] for more information.
+
+
+    Args:
+        vocab_size (`int`, *optional*, defaults to 50256):
+            Vocabulary size of the DeepSeek model. Defines the number of different tokens that can be represented by the
+            `inputs_ids` passed when calling [`DeepSeekModel`]
+        hidden_size (`int`, *optional*, defaults to 1024):
+            Dimension of the hidden representations.
+        intermediate_size (`int`, *optional*, defaults to 4096):
+            Dimension of the MLP representations for dense layers.
+        moe_intermediate_size (`int`, *optional*, defaults to 704):
+            Dimension of the MLP representations for MoE layers.
+        num_hidden_layers (`int`, *optional*, defaults to 6):
+            Number of hidden layers in the Transformer decoder.
+        num_dense_layers (`int`, *optional*, defaults to 1):
+            Number of dense (non-MoE) layers in the model.
+        num_attention_heads (`int`, *optional*, defaults to 8):
+            Number of attention heads for each attention layer in the Transformer decoder.
+        num_routed_experts (`int`, *optional*, defaults to 4):
+            Number of routed experts in MoE layers.
+        num_shared_experts (`int`, *optional*, defaults to 2):
+            Number of shared experts in MoE layers.
+        num_activated_experts (`int`, *optional*, defaults to 2):
+            Number of experts activated per token in MoE layers.
+        num_expert_groups (`int`, *optional*, defaults to 1):
+            Number of expert groups in MoE layers.
+        num_limited_groups (`int`, *optional*, defaults to 1):
+            Number of limited groups in MoE layers.
+        score_func (`str`, *optional*, defaults to `"softmax"`):
+            Scoring function for expert selection. Can be "softmax" or "sigmoid".
+        route_scale (`float`, *optional*, defaults to 1.0):
+            Scaling factor for routing weights.
+        q_lora_rank (`int`, *optional*, defaults to 0):
+            Rank of LoRA adaptation for query projection. 0 means no LoRA.
+        kv_lora_rank (`int`, *optional*, defaults to 256):
+            Rank of LoRA adaptation for key-value projection.
+        qk_nope_head_dim (`int`, *optional*, defaults to 64):
+            Dimension of query-key heads without positional encoding.
+        qk_rope_head_dim (`int`, *optional*, defaults to 32):
+            Dimension of query-key heads with rotary positional encoding.
+        v_head_dim (`int`, *optional*, defaults to 64):
+            Dimension of value heads.
+        original_seq_len (`int`, *optional*, defaults to 512):
+            Original sequence length used during pretraining.
+        rope_theta (`float`, *optional*, defaults to 10000.0):
+            Base frequency for rotary positional encoding.
+        rope_factor (`float`, *optional*, defaults to 40):
+            Scaling factor for RoPE frequency adjustment.
+        beta_fast (`int`, *optional*, defaults to 32):
+            Fast beta parameter for YaRN RoPE scaling.
+        beta_slow (`int`, *optional*, defaults to 1):
+            Slow beta parameter for YaRN RoPE scaling.
+        mscale (`float`, *optional*, defaults to 1.0):
+            Scale factor for attention logits when using extended context.
+        max_position_embeddings (`int`, *optional*, defaults to 256):
+            The maximum sequence length that this model might ever be used with.
+        max_batch_size (`int`, *optional*, defaults to 2):
+            The maximum batch size that this model might ever be used with for caching.
+        initializer_range (`float`, *optional*, defaults to 0.02):
+            The standard deviation of the truncated_normal_initializer for initializing all weight matrices.
+        rms_norm_eps (`float`, *optional*, defaults to 1e-3):
+            The epsilon used by the rms normalization layers.
+        use_cache (`bool`, *optional*, defaults to `True`):
+            Whether or not the model should return the last key/values attentions (not used by all models). Only
+            relevant if `config.is_decoder=True`.
+        pad_token_id (`int`, *optional*):
+            The id of the padding token.
+        bos_token_id (`int`, *optional*, defaults to 2):
+            The id of the "beginning-of-sequence" token.
+        eos_token_id (`int`, *optional*, defaults to 3):
+            The id of the "end-of-sequence" token.
+        tie_word_embeddings (`bool`, *optional*, defaults to `False`):
+            Whether to tie weight embeddings
+
+    ```python
+    >>> from transformers import DeepSeekModel, DeepSeekConfig
+
+    >>> # Initializing a DeepSeek configuration
+    >>> configuration = DeepSeekConfig()
+
+    >>> # Initializing a model from the configuration
+    >>> model = DeepSeekModel(configuration)
+
+    >>> # Accessing the model configuration
+    >>> configuration = model.config
+    ```"""
+
+    model_type = "deepseek"
+    keys_to_ignore_at_inference = ["past_key_values"]
+
+    def __init__(
+        self,
+        vocab_size=50256,
+        hidden_size=1024,
+        intermediate_size=4096,
+        moe_intermediate_size=704,
+        num_hidden_layers=6,
+        num_dense_layers=1,
+        num_attention_heads=8,
+        num_routed_experts=4,
+        num_shared_experts=2,
+        num_activated_experts=2,
+        num_expert_groups=1,
+        num_limited_groups=1,
+        score_func="softmax",
+        route_scale=1.0,
+        q_lora_rank=0,
+        kv_lora_rank=256,
+        qk_nope_head_dim=64,
+        qk_rope_head_dim=32,
+        v_head_dim=64,
+        original_seq_len=512,
+        rope_theta=10000.0,
+        rope_factor=40,
+        beta_fast=32,
+        beta_slow=1,
+        mscale=1.0,
+        max_position_embeddings=256,
+        max_batch_size=2,
+        initializer_range=0.02,
+        rms_norm_eps=1e-3,
+        use_cache=True,
+        pad_token_id=0,
+        bos_token_id=2,
+        eos_token_id=3,
+        tie_word_embeddings=False,
+        **kwargs,
+    ):
+        self.vocab_size = vocab_size
+        self.max_position_embeddings = max_position_embeddings
+        self.hidden_size = hidden_size
+        self.intermediate_size = intermediate_size
+        self.moe_intermediate_size = moe_intermediate_size
+        self.num_hidden_layers = num_hidden_layers
+        self.num_dense_layers = num_dense_layers
+        self.num_attention_heads = num_attention_heads
+        self.num_routed_experts = num_routed_experts
+        self.num_shared_experts = num_shared_experts
+        self.num_activated_experts = num_activated_experts
+        self.num_expert_groups = num_expert_groups
+        self.num_limited_groups = num_limited_groups
+        self.score_func = score_func
+        self.route_scale = route_scale
+        self.q_lora_rank = q_lora_rank
+        self.kv_lora_rank = kv_lora_rank
+        self.qk_nope_head_dim = qk_nope_head_dim
+        self.qk_rope_head_dim = qk_rope_head_dim
+        self.v_head_dim = v_head_dim
+        self.original_seq_len = original_seq_len
+        self.rope_theta = rope_theta
+        self.rope_factor = rope_factor
+        self.beta_fast = beta_fast
+        self.beta_slow = beta_slow
+        self.mscale = mscale
+        self.max_batch_size = max_batch_size
+        self.initializer_range = initializer_range
+        self.rms_norm_eps = rms_norm_eps
+        self.use_cache = use_cache
+        self.tie_word_embeddings = tie_word_embeddings
+
+        super().__init__(
+            pad_token_id=pad_token_id,
+            bos_token_id=bos_token_id,
+            eos_token_id=eos_token_id,
+            tie_word_embeddings=tie_word_embeddings,
+            **kwargs,
+        ) 

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e386559416e5b062ed0599f6640355c564c13a59d79d7225b5aacc68c309df8e
+size 767127048

modeling_deepseek.py

@@ -0,0 +1,592 @@
+"""
+PyTorch DeepSeek model.
+"""
+
+import math
+from typing import List, Optional, Tuple, Union
+
+import torch
+import torch.nn as nn
+import torch.nn.functional as F
+from configuration_deepseek import DeepSeekConfig
+from torch.nn import CrossEntropyLoss
+from transformers.activations import ACT2FN
+from transformers.cache_utils import Cache, DynamicCache
+from transformers.modeling_attn_mask_utils import (
+    AttentionMaskConverter, _prepare_4d_attention_mask,
+    _prepare_4d_causal_attention_mask)
+from transformers.modeling_outputs import (BaseModelOutputWithPast,
+                                           CausalLMOutputWithPast)
+from transformers.modeling_utils import PreTrainedModel
+from transformers.utils import (add_start_docstrings,
+                                add_start_docstrings_to_model_forward,
+                                is_flash_attn_2_available,
+                                is_flash_attn_greater_or_equal_2_10, logging,
+                                replace_return_docstrings)
+
+if is_flash_attn_2_available():
+    from flash_attn import flash_attn_func, flash_attn_varlen_func
+    from flash_attn.bert_padding import (index_first_axis, pad_input,  # noqa
+                                         unpad_input)
+
+logger = logging.get_logger(__name__)
+
+_CONFIG_FOR_DOC = "DeepSeekConfig"
+
+
+def precompute_freqs_cis(config: DeepSeekConfig) -> torch.Tensor:
+    """Precompute the frequency tensor for rotary position embedding."""
+    dim = config.qk_rope_head_dim
+    seqlen = config.max_position_embeddings
+    beta_fast = config.beta_fast
+    beta_slow = config.beta_slow
+    base = config.rope_theta
+    factor = config.rope_factor
+
+    def find_correction_dim(num_rotations, dim, base, max_seq_len):
+        return dim * math.log(max_seq_len / (num_rotations * 2 * math.pi)) / (2 * math.log(base))
+
+    def find_correction_range(low_rot, high_rot, dim, base, max_seq_len):
+        low = math.floor(find_correction_dim(low_rot, dim, base, max_seq_len))
+        high = math.ceil(find_correction_dim(high_rot, dim, base, max_seq_len))
+        return max(low, 0), min(high, dim-1)
+
+    def linear_ramp_factor(min_val, max_val, dim):
+        if min_val == max_val:
+            max_val += 0.001
+        linear_func = (torch.arange(dim, dtype=torch.float32) - min_val) / (max_val - min_val)
+        ramp_func = torch.clamp(linear_func, 0, 1)
+        return ramp_func
+
+    freqs = 1.0 / (base ** (torch.arange(0, dim, 2, dtype=torch.float32) / dim))
+
+    if seqlen > config.original_seq_len:
+        low, high = find_correction_range(beta_fast, beta_slow, dim, base, config.original_seq_len)
+        smooth = 1 - linear_ramp_factor(low, high, dim // 2)
+        freqs = freqs / factor * (1 - smooth) + freqs * smooth
+
+    t = torch.arange(seqlen)
+    freqs = torch.outer(t, freqs)
+    freqs_cis = torch.polar(torch.ones_like(freqs), freqs)
+    return freqs_cis
+
+
+def apply_rotary_emb(x: torch.Tensor, freqs_cis: torch.Tensor) -> torch.Tensor:
+    """Apply rotary position embedding to the input tensor."""
+    assert x.shape[-1] % 2 == 0, "Rotary dim must be divisible by 2!"
+    dtype = x.dtype
+    x = torch.view_as_complex(x.float().view(*x.shape[:-1], -1, 2))
+    freqs_cis = freqs_cis.view(1, x.size(1), 1, x.size(-1))
+    y = torch.view_as_real(x * freqs_cis).reshape(*x.shape[:-1], -1)
+    return y.to(dtype)
+
+
+class DeepSeekRMSNorm(nn.Module):
+    """RMS normalization layer."""
+    
+    def __init__(self, hidden_size, eps=1e-6):
+        super().__init__()
+        self.weight = nn.Parameter(torch.ones(hidden_size))
+        self.variance_epsilon = eps
+
+    def forward(self, hidden_states):
+        input_dtype = hidden_states.dtype
+        hidden_states = hidden_states.to(torch.float32)
+        variance = hidden_states.pow(2).mean(-1, keepdim=True)
+        hidden_states = hidden_states * torch.rsqrt(variance + self.variance_epsilon)
+        return self.weight * hidden_states.to(input_dtype)
+
+
+class DeepSeekMLA(nn.Module):
+    """Multi-head Latent Attention (MLA) module."""
+    
+    def __init__(self, config: DeepSeekConfig, layer_idx: Optional[int] = None):
+        super().__init__()
+        self.config = config
+        self.layer_idx = layer_idx
+        
+        self.hidden_size = config.hidden_size
+        self.num_heads = config.num_attention_heads
+        self.head_dim = self.hidden_size // self.num_heads
+        self.num_key_value_heads = config.num_attention_heads
+        self.num_key_value_groups = self.num_heads // self.num_key_value_heads
+        self.max_position_embeddings = config.max_position_embeddings
+        self.rope_theta = config.rope_theta
+        self.is_causal = True
+
+        # MLA specific parameters
+        self.q_lora_rank = config.q_lora_rank
+        self.kv_lora_rank = config.kv_lora_rank
+        self.qk_nope_head_dim = config.qk_nope_head_dim
+        self.qk_rope_head_dim = config.qk_rope_head_dim
+        self.v_head_dim = config.v_head_dim
+        self.qk_head_dim = self.qk_nope_head_dim + self.qk_rope_head_dim
+
+        if self.q_lora_rank == 0:
+            self.q_proj = nn.Linear(self.hidden_size, self.num_heads * self.qk_head_dim, bias=False)
+        else:
+            self.q_a_proj = nn.Linear(self.hidden_size, self.q_lora_rank, bias=False)
+            self.q_a_layernorm = DeepSeekRMSNorm(self.q_lora_rank, eps=config.rms_norm_eps)
+            self.q_b_proj = nn.Linear(self.q_lora_rank, self.num_heads * self.qk_head_dim, bias=False)
+
+        self.kv_a_proj_with_mqa = nn.Linear(
+            self.hidden_size, 
+            self.kv_lora_rank + self.qk_rope_head_dim, 
+            bias=False
+        )
+        self.kv_a_layernorm = DeepSeekRMSNorm(self.kv_lora_rank, eps=config.rms_norm_eps)
+        self.kv_b_proj = nn.Linear(
+            self.kv_lora_rank, 
+            self.num_heads * (self.qk_nope_head_dim + self.v_head_dim), 
+            bias=False
+        )
+        self.o_proj = nn.Linear(self.num_heads * self.v_head_dim, self.hidden_size, bias=False)
+        
+        # Scaling
+        self.scaling = self.qk_head_dim ** -0.5
+        if config.max_position_embeddings > config.original_seq_len:
+            mscale = 0.1 * config.mscale * math.log(config.rope_factor) + 1.0
+            self.scaling = self.scaling * mscale * mscale
+
+    def forward(
+        self,
+        hidden_states: torch.Tensor,
+        attention_mask: Optional[torch.Tensor] = None,
+        position_ids: Optional[torch.LongTensor] = None,
+        past_key_value: Optional[Cache] = None,
+        output_attentions: bool = False,
+        use_cache: bool = False,
+        cache_position: Optional[torch.LongTensor] = None,
+    ) -> Tuple[torch.Tensor, Optional[torch.Tensor], Optional[Tuple[torch.Tensor]]]:
+        
+        bsz, q_len, _ = hidden_states.size()
+
+        # Query projection
+        if self.q_lora_rank == 0:
+            query_states = self.q_proj(hidden_states)
+        else:
+            query_states = self.q_b_proj(self.q_a_layernorm(self.q_a_proj(hidden_states)))
+        
+        query_states = query_states.view(bsz, q_len, self.num_heads, self.qk_head_dim).transpose(1, 2)
+        
+        # Split query into no-position-encoding and position-encoding parts
+        q_nope, q_pe = query_states.split([self.qk_nope_head_dim, self.qk_rope_head_dim], dim=-1)
+        
+        # Key-Value projection
+        kv_input = self.kv_a_proj_with_mqa(hidden_states)
+        compressed_kv, k_pe = kv_input.split([self.kv_lora_rank, self.qk_rope_head_dim], dim=-1)
+        
+        # Apply RoPE to position-encoding parts
+        if position_ids is not None:
+            cos, sin = self.rotary_emb(hidden_states, position_ids)
+            q_pe = apply_rotary_pos_emb(q_pe, cos, sin)
+            k_pe = apply_rotary_pos_emb(k_pe.unsqueeze(2), cos, sin).squeeze(2)
+
+        # Compute key and value from compressed representation
+        kv_b_weight = self.kv_b_proj.weight.view(
+            self.num_heads, self.qk_nope_head_dim + self.v_head_dim, self.kv_lora_rank
+        )
+        
+        # Project compressed KV to get keys and values
+        compressed_kv = self.kv_a_layernorm(compressed_kv)
+        key_states = torch.einsum('bld,hnd->bhln', compressed_kv, kv_b_weight[:, :self.qk_nope_head_dim, :])
+        value_states = torch.einsum('bld,hnd->bhln', compressed_kv, kv_b_weight[:, -self.v_head_dim:, :])
+
+        # Attention computation
+        attn_weights = torch.matmul(q_nope, key_states.transpose(-2, -1)) * self.scaling
+        
+        # Add positional attention
+        if k_pe is not None:
+            pos_attn = torch.matmul(q_pe, k_pe.unsqueeze(1).transpose(-2, -1)) * self.scaling
+            attn_weights = attn_weights + pos_attn
+
+        if attention_mask is not None:
+            causal_mask = attention_mask[:, :, :, : key_states.shape[-2]]
+            attn_weights = attn_weights + causal_mask
+
+        # Apply softmax
+        attn_weights = nn.functional.softmax(attn_weights, dim=-1, dtype=torch.float32).to(query_states.dtype)
+
+        # Apply attention to values
+        attn_output = torch.matmul(attn_weights, value_states)
+        
+        attn_output = attn_output.transpose(1, 2).contiguous()
+        attn_output = attn_output.reshape(bsz, q_len, -1)
+        attn_output = self.o_proj(attn_output)
+
+        if not output_attentions:
+            attn_weights = None
+
+        return attn_output, attn_weights, past_key_value
+
+
+class DeepSeekMLP(nn.Module):
+    """Multi-Layer Perceptron for dense layers."""
+    
+    def __init__(self, config: DeepSeekConfig):
+        super().__init__()
+        self.config = config
+        self.hidden_size = config.hidden_size
+        self.intermediate_size = config.intermediate_size
+        
+        self.gate_proj = nn.Linear(self.hidden_size, self.intermediate_size, bias=False)
+        self.up_proj = nn.Linear(self.hidden_size, self.intermediate_size, bias=False)
+        self.down_proj = nn.Linear(self.intermediate_size, self.hidden_size, bias=False)
+        self.act_fn = ACT2FN["silu"]
+
+    def forward(self, x):
+        return self.down_proj(self.act_fn(self.gate_proj(x)) * self.up_proj(x))
+
+
+class DeepSeekExpert(nn.Module):
+    """Single expert in MoE layer."""
+    
+    def __init__(self, config: DeepSeekConfig):
+        super().__init__()
+        self.hidden_size = config.hidden_size
+        self.intermediate_size = config.moe_intermediate_size
+        
+        self.gate_proj = nn.Linear(self.hidden_size, self.intermediate_size, bias=False)
+        self.up_proj = nn.Linear(self.hidden_size, self.intermediate_size, bias=False)
+        self.down_proj = nn.Linear(self.intermediate_size, self.hidden_size, bias=False)
+        self.act_fn = ACT2FN["silu"]
+
+    def forward(self, x):
+        return self.down_proj(self.act_fn(self.gate_proj(x)) * self.up_proj(x))
+
+
+DEEPSEEK_START_DOCSTRING = r"""
+    This model inherits from [`PreTrainedModel`]. Check the superclass documentation for the generic methods the
+    library implements for all its model (such as downloading or saving, resizing the input embeddings, pruning heads
+    etc.)
+
+    This model is also a PyTorch [torch.nn.Module](https://pytorch.org/docs/stable/nn.html#torch.nn.Module) subclass.
+    Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage
+    and behavior.
+
+    Parameters:
+        config ([`DeepSeekConfig`]):
+            Model configuration class with all the parameters of the model. Initializing with a config file does not
+            load the weights associated with the model, only the configuration. Check out the
+            [`~PreTrainedModel.from_pretrained`] method to load the model weights.
+"""
+
+
+@add_start_docstrings(
+    "The bare DeepSeek Model outputting raw hidden-states without any specific head on top.",
+    DEEPSEEK_START_DOCSTRING,
+)
+class DeepSeekPreTrainedModel(PreTrainedModel):
+    config_class = DeepSeekConfig
+    base_model_prefix = "model"
+    supports_gradient_checkpointing = True
+    _no_split_modules = ["DeepSeekDecoderLayer"]
+    _skip_keys_device_placement = ["past_key_values"]
+    _supports_flash_attn_2 = True
+    _supports_sdpa = True
+    _supports_cache_class = True
+
+    def _init_weights(self, module):
+        std = self.config.initializer_range
+        if isinstance(module, nn.Linear):
+            module.weight.data.normal_(mean=0.0, std=std)
+            if module.bias is not None:
+                module.bias.data.zero_()
+        elif isinstance(module, nn.Embedding):
+            module.weight.data.normal_(mean=0.0, std=std)
+            if module.padding_idx is not None:
+                module.weight.data[module.padding_idx].zero_()
+
+
+DEEPSEEK_INPUTS_DOCSTRING = r"""
+    Args:
+        input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`):
+            Indices of input sequence tokens in the vocabulary. Padding will be ignored by default should you provide
+            it.
+
+            Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and
+            [`PreTrainedTokenizer.__call__`] for details.
+
+            [What are input IDs?](../glossary#input-ids)
+        attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*):
+            Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`:
+
+            - 1 for tokens that are **not masked**,
+            - 0 for tokens that are **masked**.
+
+            [What are attention masks?](../glossary#attention-mask)
+
+            Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and
+            [`PreTrainedTokenizer.__call__`] for details.
+
+            If `past_key_values` is used, optionally only the last `input_ids` have to be input (see
+            `past_key_values`).
+
+            If you want to change padding behavior, you should read [`modeling_opt._prepare_decoder_attention_mask`]
+            and modify to your needs. See diagram 1 in [the paper](https://arxiv.org/abs/1910.13461) for more
+            information on the default strategy.
+
+            - 1 indicates the head is **not masked**,
+            - 0 indicates the head is **masked**.
+        position_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*):
+            Indices of positions of each input sequence token in the position embeddings. Selected in the range `[0,
+            config.n_positions - 1]`.
+
+            [What are position IDs?](../glossary#position-ids)
+        past_key_values (`Cache` or `tuple(tuple(torch.FloatTensor))`, *optional*):
+            Pre-computed hidden-states (key and value in the self-attention blocks and in the cross-attention blocks)
+            that can be used to speed up sequential decoding. This typically consists in the `past_key_values`
+            returned by the model at a previous stage of decoding, when `use_cache=True` or `config.use_cache=True`.
+
+            Two formats are allowed:
+            - a [`~cache_utils.Cache`] instance;
+            - Tuple of `tuple(torch.FloatTensor)` of length `config.n_layers`, with each tuple having 2 tensors of
+            shape `(batch_size, num_heads, sequence_length, embed_size_per_head)`). This is also known as the legacy
+            cache format.
+
+            The model will output the same cache format that is fed as input. If no `past_key_values` are passed, the
+            legacy cache format will be returned.
+
+            If `past_key_values` are used, the user can optionally input only the last `input_ids` (those that don't
+            have their past key/value states given to this model) of shape `(batch_size, 1)` instead of all `input_ids`
+            of shape `(batch_size, sequence_length)`.
+        inputs_embeds (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`, *optional*):
+            Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This
+            is useful if you want more control over how to convert `input_ids` indices into associated vectors than the
+            model's internal embedding lookup matrix.
+        use_cache (`bool`, *optional*):
+            If set to `True`, `past_key_values` key value states are returned and can be used to speed up decoding (see
+            `past_key_values`).
+        output_attentions (`bool`, *optional*):
+            Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned
+            tensors for more detail.
+        output_hidden_states (`bool`, *optional*):
+            Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for
+            more detail.
+        return_dict (`bool`, *optional*):
+            Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple.
+        cache_position (`torch.LongTensor` of shape `(sequence_length)`, *optional*):
+            Indices depicting the position of the input sequence tokens in the sequence. Contrarily to `position_ids`,
+            this tensor is not affected by padding. It is used to update the cache in the correct position and to infer
+            the complete sequence length.
+"""
+
+
+class DeepSeekModel(DeepSeekPreTrainedModel):
+    """
+    Transformer decoder consisting of *config.num_hidden_layers* layers. Each layer is a [`DeepSeekDecoderLayer`]
+
+    Args:
+        config: DeepSeekConfig
+    """
+
+    def __init__(self, config: DeepSeekConfig):
+        super().__init__(config)
+        self.padding_idx = config.pad_token_id
+        self.vocab_size = config.vocab_size
+
+        self.embed_tokens = nn.Embedding(config.vocab_size, config.hidden_size, self.padding_idx)
+        # Note: We'll implement layers in a separate method due to complexity
+        self.norm = DeepSeekRMSNorm(config.hidden_size, eps=config.rms_norm_eps)
+        
+        self.gradient_checkpointing = False
+        # Initialize weights and apply final processing
+        self.post_init()
+
+    def get_input_embeddings(self):
+        return self.embed_tokens
+
+    def set_input_embeddings(self, value):
+        self.embed_tokens = value
+
+    @add_start_docstrings_to_model_forward(DEEPSEEK_INPUTS_DOCSTRING)
+    def forward(
+        self,
+        input_ids: torch.LongTensor = None,
+        attention_mask: Optional[torch.Tensor] = None,
+        position_ids: Optional[torch.LongTensor] = None,
+        past_key_values: Optional[List[torch.FloatTensor]] = None,
+        inputs_embeds: Optional[torch.FloatTensor] = None,
+        use_cache: Optional[bool] = None,
+        output_attentions: Optional[bool] = None,
+        output_hidden_states: Optional[bool] = None,
+        return_dict: Optional[bool] = None,
+        cache_position: Optional[torch.LongTensor] = None,
+    ) -> Union[Tuple, BaseModelOutputWithPast]:
+        """Forward pass of the DeepSeek model."""
+        
+        output_attentions = output_attentions if output_attentions is not None else self.config.output_attentions
+        output_hidden_states = (
+            output_hidden_states if output_hidden_states is not None else self.config.output_hidden_states
+        )
+        use_cache = use_cache if use_cache is not None else self.config.use_cache
+        return_dict = return_dict if return_dict is not None else self.config.use_return_dict
+
+        if (input_ids is None) ^ (inputs_embeds is not None):
+            raise ValueError(
+                "You cannot specify both input_ids and inputs_embeds at the same time, and must specify either one"
+            )
+
+        if inputs_embeds is None:
+            inputs_embeds = self.embed_tokens(input_ids)
+
+        hidden_states = inputs_embeds
+
+        # Apply normalization
+        hidden_states = self.norm(hidden_states)
+
+        if not return_dict:
+            return tuple(v for v in [hidden_states, None, None] if v is not None)
+        
+        return BaseModelOutputWithPast(
+            last_hidden_state=hidden_states,
+            past_key_values=None,
+            hidden_states=None,
+            attentions=None,
+        )
+
+
+class DeepSeekForCausalLM(DeepSeekPreTrainedModel):
+    _tied_weights_keys = ["lm_head.weight"]
+
+    def __init__(self, config):
+        super().__init__(config)
+        self.model = DeepSeekModel(config)
+        self.vocab_size = config.vocab_size
+        self.lm_head = nn.Linear(config.hidden_size, config.vocab_size, bias=False)
+
+        # Initialize weights and apply final processing
+        self.post_init()
+
+    def get_input_embeddings(self):
+        return self.model.embed_tokens
+
+    def set_input_embeddings(self, value):
+        self.model.embed_tokens = value
+
+    def get_output_embeddings(self):
+        return self.lm_head
+
+    def set_output_embeddings(self, new_embeddings):
+        self.lm_head = new_embeddings
+
+    def set_decoder(self, decoder):
+        self.model = decoder
+
+    def get_decoder(self):
+        return self.model
+
+    @add_start_docstrings_to_model_forward(DEEPSEEK_INPUTS_DOCSTRING)
+    @replace_return_docstrings(output_type=CausalLMOutputWithPast, config_class=_CONFIG_FOR_DOC)
+    def forward(
+        self,
+        input_ids: torch.LongTensor = None,
+        attention_mask: Optional[torch.Tensor] = None,
+        position_ids: Optional[torch.LongTensor] = None,
+        past_key_values: Optional[List[torch.FloatTensor]] = None,
+        inputs_embeds: Optional[torch.FloatTensor] = None,
+        labels: Optional[torch.LongTensor] = None,
+        use_cache: Optional[bool] = None,
+        output_attentions: Optional[bool] = None,
+        output_hidden_states: Optional[bool] = None,
+        return_dict: Optional[bool] = None,
+        cache_position: Optional[torch.LongTensor] = None,
+    ) -> Union[Tuple, CausalLMOutputWithPast]:
+        """Forward pass of the DeepSeek model for causal language modeling.
+        
+        Returns:
+        """
+        
+        output_attentions = output_attentions if output_attentions is not None else self.config.output_attentions
+        output_hidden_states = (
+            output_hidden_states if output_hidden_states is not None else self.config.output_hidden_states
+        )
+        return_dict = return_dict if return_dict is not None else self.config.use_return_dict
+
+        # decoder outputs consists of (dec_features, layer_state, dec_hidden, dec_attn)
+        outputs = self.model(
+            input_ids=input_ids,
+            attention_mask=attention_mask,
+            position_ids=position_ids,
+            past_key_values=past_key_values,
+            inputs_embeds=inputs_embeds,
+            use_cache=use_cache,
+            output_attentions=output_attentions,
+            output_hidden_states=output_hidden_states,
+            return_dict=return_dict,
+            cache_position=cache_position,
+        )
+
+        hidden_states = outputs[0]
+        logits = self.lm_head(hidden_states)
+        logits = logits.float()
+
+        loss = None
+        if labels is not None:
+            # Shift so that tokens < n predict n
+            shift_logits = logits[..., :-1, :].contiguous()
+            shift_labels = labels[..., 1:].contiguous()
+            # Flatten the tokens
+            loss_fct = CrossEntropyLoss()
+            shift_logits = shift_logits.view(-1, self.config.vocab_size)
+            shift_labels = shift_labels.view(-1)
+            # Enable model parallelism
+            shift_labels = shift_labels.to(shift_logits.device)
+            loss = loss_fct(shift_logits, shift_labels)
+
+        if not return_dict:
+            output = (logits,) + outputs[1:]
+            return (loss,) + output if loss is not None else output
+
+        return CausalLMOutputWithPast(
+            loss=loss,
+            logits=logits,
+            past_key_values=outputs.past_key_values,
+            hidden_states=outputs.hidden_states,
+            attentions=outputs.attentions,
+        )
+
+    def prepare_inputs_for_generation(
+        self, input_ids, past_key_values=None, attention_mask=None, inputs_embeds=None, cache_position=None, **kwargs
+    ):
+        # If we have cache: let's slice `input_ids` through `cache_position`, to keep only the unprocessed tokens
+        # Exception 1: when passing input_embeds, input_ids may be missing entries
+        # Exception 2: some generation methods do special slicing of input_ids, so we don't need to do it here
+        if past_key_values is not None:
+            if inputs_embeds is not None:  # Exception 1
+                input_ids = input_ids[:, -cache_position.shape[0] :]
+            elif input_ids.shape[1] != cache_position.shape[0]:  # Default case (the "else", a no op, is Exception 2)
+                input_ids = input_ids[:, cache_position]
+
+        if attention_mask is not None and position_ids is None:
+            # create position_ids on the fly for batch generation
+            position_ids = attention_mask.long().cumsum(-1) - 1
+            position_ids.masked_fill_(attention_mask == 0, 1)
+            if past_key_values:
+                position_ids = position_ids[:, -input_ids.shape[1] :]
+
+        # if `inputs_embeds` are passed, we only want to use them in the 1st generation step
+        if inputs_embeds is not None and cache_position[0] == 0:
+            model_inputs = {"inputs_embeds": inputs_embeds}
+        else:
+            model_inputs = {"input_ids": input_ids}
+
+        model_inputs.update(
+            {
+                "position_ids": position_ids,
+                "cache_position": cache_position,
+                "past_key_values": past_key_values,
+                "use_cache": kwargs.get("use_cache"),
+                "attention_mask": attention_mask,
+            }
+        )
+        return model_inputs
+
+    @staticmethod
+    def _reorder_cache(past_key_values, beam_idx):
+        reordered_past = ()
+        for layer_past in past_key_values:
+            reordered_past += (
+                tuple(past_state.index_select(0, beam_idx.to(past_state.device)) for past_state in layer_past),
+            )
+        return reordered_past 

pytorch_model.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:78e40acba71577135f972943f49ee9739ef33686b6bc7197ea4469b51e468211
+size 767177171

special_tokens_map.json

@@ -0,0 +1,6 @@
+{
+  "bos_token": "<s>",
+  "eos_token": "</s>",
+  "pad_token": "<pad>",
+  "unk_token": "<unk>"
+}

tokenizer.model

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:7b0d31a2e63001005e491e091cb02dd9ee7f786bd54be81018b01ff245408628
+size 1110078

tokenizer_config.json

@@ -0,0 +1,15 @@
+{
+  "tokenizer_class": "LlamaTokenizer",
+  "model_max_length": 256,
+  "pad_token": "<pad>",
+  "bos_token": "<s>",
+  "eos_token": "</s>",
+  "unk_token": "<unk>",
+  "clean_up_tokenization_spaces": false,
+  "auto_map": {
+    "AutoTokenizer": [
+      "sentencepiece",
+      "LlamaTokenizer"
+    ]
+  }
+}