Upload MistsForConditionalGeneration

README.md

@@ -0,0 +1,199 @@
+---
+library_name: transformers
+tags: []
+---
+
+# Model Card for Model ID
+
+<!-- Provide a quick summary of what the model is/does. -->
+
+
+
+## Model Details
+
+### Model Description
+
+<!-- Provide a longer summary of what this model is. -->
+
+This is the model card of a 🤗 transformers model that has been pushed on the Hub. This model card has been automatically generated.
+
+- **Developed by:** [More Information Needed]
+- **Funded by [optional]:** [More Information Needed]
+- **Shared by [optional]:** [More Information Needed]
+- **Model type:** [More Information Needed]
+- **Language(s) (NLP):** [More Information Needed]
+- **License:** [More Information Needed]
+- **Finetuned from model [optional]:** [More Information Needed]
+
+### Model Sources [optional]
+
+<!-- Provide the basic links for the model. -->
+
+- **Repository:** [More Information Needed]
+- **Paper [optional]:** [More Information Needed]
+- **Demo [optional]:** [More Information Needed]
+
+## Uses
+
+<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
+
+### Direct Use
+
+<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
+
+[More Information Needed]
+
+### Downstream Use [optional]
+
+<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
+
+[More Information Needed]
+
+### Out-of-Scope Use
+
+<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
+
+[More Information Needed]
+
+## Bias, Risks, and Limitations
+
+<!-- This section is meant to convey both technical and sociotechnical limitations. -->
+
+[More Information Needed]
+
+### Recommendations
+
+<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
+
+Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
+
+## How to Get Started with the Model
+
+Use the code below to get started with the model.
+
+[More Information Needed]
+
+## Training Details
+
+### Training Data
+
+<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
+
+[More Information Needed]
+
+### Training Procedure
+
+<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
+
+#### Preprocessing [optional]
+
+[More Information Needed]
+
+
+#### Training Hyperparameters
+
+- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
+
+#### Speeds, Sizes, Times [optional]
+
+<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
+
+[More Information Needed]
+
+## Evaluation
+
+<!-- This section describes the evaluation protocols and provides the results. -->
+
+### Testing Data, Factors & Metrics
+
+#### Testing Data
+
+<!-- This should link to a Dataset Card if possible. -->
+
+[More Information Needed]
+
+#### Factors
+
+<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
+
+[More Information Needed]
+
+#### Metrics
+
+<!-- These are the evaluation metrics being used, ideally with a description of why. -->
+
+[More Information Needed]
+
+### Results
+
+[More Information Needed]
+
+#### Summary
+
+
+
+## Model Examination [optional]
+
+<!-- Relevant interpretability work for the model goes here -->
+
+[More Information Needed]
+
+## Environmental Impact
+
+<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
+
+Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
+
+- **Hardware Type:** [More Information Needed]
+- **Hours used:** [More Information Needed]
+- **Cloud Provider:** [More Information Needed]
+- **Compute Region:** [More Information Needed]
+- **Carbon Emitted:** [More Information Needed]
+
+## Technical Specifications [optional]
+
+### Model Architecture and Objective
+
+[More Information Needed]
+
+### Compute Infrastructure
+
+[More Information Needed]
+
+#### Hardware
+
+[More Information Needed]
+
+#### Software
+
+[More Information Needed]
+
+## Citation [optional]
+
+<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
+
+**BibTeX:**
+
+[More Information Needed]
+
+**APA:**
+
+[More Information Needed]
+
+## Glossary [optional]
+
+<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
+
+[More Information Needed]
+
+## More Information [optional]
+
+[More Information Needed]
+
+## Model Card Authors [optional]
+
+[More Information Needed]
+
+## Model Card Contact
+
+[More Information Needed]

config.json

@@ -0,0 +1,126 @@
+{
+  "architectures": [
+    "MistsForConditionalGeneration"
+  ],
+  "auto_map": {
+    "AutoConfig": "configuration_mists.MistsConfig",
+    "AutoModel": "modeling_mists.MistsForConditionalGeneration"
+  },
+  "ignore_index": -100,
+  "model_type": "mists",
+  "pad_token_id": 32769,
+  "projector_hidden_act": "gelu",
+  "text_config": {
+    "_name_or_path": "mistralai/Mistral-7B-Instruct-v0.3",
+    "architectures": [
+      "MistralForCausalLM"
+    ],
+    "max_position_embeddings": 32768,
+    "model_type": "mistral",
+    "rms_norm_eps": 1e-05,
+    "rope_theta": 1000000.0,
+    "sliding_window": null,
+    "torch_dtype": "bfloat16",
+    "vocab_size": 32832
+  },
+  "time_series_config": {
+    "_name_or_path": "HachiML/MOMENT-1-large-embedding-v0.1",
+    "architectures": [
+      "MomentEmbeddingModel"
+    ],
+    "auto_map": {
+      "AutoConfig": "HachiML/MOMENT-1-large-embedding-v0.1--configuration_moment.MomentConfig",
+      "AutoModel": "HachiML/MOMENT-1-large-embedding-v0.1--modeling_moment.MomentEmbeddingModel"
+    },
+    "mask_ratio": 0.0,
+    "model_type": "moment",
+    "patch_len": 8,
+    "patch_stride_len": 8,
+    "revin_affine": false,
+    "t5_config": {
+      "add_cross_attention": false,
+      "attn_implementation": null,
+      "bad_words_ids": null,
+      "begin_suppress_tokens": null,
+      "bos_token_id": null,
+      "chunk_size_feed_forward": 0,
+      "classifier_dropout": 0.0,
+      "cross_attention_hidden_size": null,
+      "d_ff": 2816,
+      "d_kv": 64,
+      "d_model": 1024,
+      "decoder_start_token_id": 0,
+      "dense_act_fn": "gelu_new",
+      "diversity_penalty": 0.0,
+      "do_sample": false,
+      "dropout_rate": 0.1,
+      "early_stopping": false,
+      "encoder_no_repeat_ngram_size": 0,
+      "eos_token_id": 1,
+      "exponential_decay_length_penalty": null,
+      "feed_forward_proj": "gated-gelu",
+      "finetuning_task": null,
+      "forced_bos_token_id": null,
+      "forced_eos_token_id": null,
+      "id2label": {
+        "0": "LABEL_0",
+        "1": "LABEL_1"
+      },
+      "initializer_factor": 1.0,
+      "is_decoder": false,
+      "is_encoder_decoder": true,
+      "is_gated_act": true,
+      "label2id": {
+        "LABEL_0": 0,
+        "LABEL_1": 1
+      },
+      "layer_norm_epsilon": 1e-06,
+      "length_penalty": 1.0,
+      "max_length": 20,
+      "min_length": 0,
+      "n_positions": 512,
+      "no_repeat_ngram_size": 0,
+      "num_beam_groups": 1,
+      "num_beams": 1,
+      "num_decoder_layers": 24,
+      "num_heads": 16,
+      "num_layers": 24,
+      "num_return_sequences": 1,
+      "output_attentions": false,
+      "output_hidden_states": false,
+      "output_past": true,
+      "output_scores": false,
+      "pad_token_id": 0,
+      "prefix": null,
+      "problem_type": null,
+      "pruned_heads": {},
+      "relative_attention_max_distance": 128,
+      "relative_attention_num_buckets": 32,
+      "remove_invalid_values": false,
+      "repetition_penalty": 1.0,
+      "return_dict": true,
+      "return_dict_in_generate": false,
+      "sep_token_id": null,
+      "suppress_tokens": null,
+      "task_specific_params": null,
+      "temperature": 1.0,
+      "tf_legacy_loss": false,
+      "tie_encoder_decoder": false,
+      "tie_word_embeddings": false,
+      "tokenizer_class": null,
+      "top_k": 50,
+      "top_p": 1.0,
+      "torch_dtype": null,
+      "torchscript": false,
+      "typical_p": 1.0,
+      "use_bfloat16": false,
+      "use_cache": true,
+      "vocab_size": 32128
+    },
+    "torch_dtype": "float32"
+  },
+  "time_series_hidden_size": 1024,
+  "time_series_token_index": 32768,
+  "torch_dtype": "float32",
+  "transformers_version": "4.41.2"
+}

configuration_mists.py

@@ -0,0 +1,60 @@
+import warnings
+
+from transformers import PretrainedConfig
+from transformers import CONFIG_MAPPING
+
+from .configuration_moment import MomentConfig
+
+class MistsConfig(PretrainedConfig):
+    model_type = "mists"
+
+    def __init__(
+        self, 
+        time_series_config=None,
+        text_config=None,
+        ignore_index=-100,
+        time_series_token_index=32000,
+        projector_hidden_act="gelu",  # projector用
+        # time_series_feature_select_strategy="default",  # TODO: modelのforward用(画像モデルのhidden_stateからEmbeddingをどう取得するか)。将来的に対応。
+        # time_series_feature_layer=-2,  # modelのforward用  # TODO: modelのforward用(画像モデルのhidden_stateからEmbeddingをどう取得するか)。将来的に対応。
+        time_series_hidden_size=1024,  # projector用
+        **kwargs,
+    ):
+        
+        self.ignore_index = ignore_index
+        self.time_series_token_index = time_series_token_index
+        self.projector_hidden_act = projector_hidden_act
+        self.time_series_hidden_size = time_series_hidden_size
+
+        # 将来的に、MomentモデルがTransformersに登録されることを想定して追加する
+        # そのため、CONFIG_MAPPINGは機能しない。
+        if isinstance(time_series_config, dict):
+            time_series_config["model_type"] = (
+                time_series_config["model_type"] if "model_type" in time_series_config else "moment"
+            )
+            # time_series_config = CONFIG_MAPPING[time_series_config["model_type"]](**time_series_config)
+            time_series_config = MomentConfig(**time_series_config)
+        elif time_series_config is None:
+            time_series_config = MomentConfig()
+
+        self.time_series_config = time_series_config
+
+        if isinstance(text_config, dict):
+            text_config["model_type"] = text_config["model_type"] if "model_type" in text_config else "mistral"
+            text_config = CONFIG_MAPPING[text_config["model_type"]](**text_config)
+        elif text_config is None:
+            text_config = CONFIG_MAPPING["mistral"]()
+
+        self.text_config = text_config
+
+        super().__init__(**kwargs)
+
+
+    def to_dict(self):
+        output = super().to_dict()
+        return output
+
+
+
+        
+

configuration_moment.py

@@ -0,0 +1,103 @@
+"""Moment model configuration"""
+
+from transformers import PretrainedConfig
+from transformers import logging
+
+
+DEFAULT_T5_CONFIG = {
+    # "_name_or_path": "google/flan-t5-large",
+    # "architectures": [
+    #     "T5ForConditionalGeneration"
+    # ],
+    "classifier_dropout": 0.0,
+    "d_ff": 2816,
+    "d_kv": 64,
+    "d_model": 1024,
+    "decoder_start_token_id": 0,
+    "dense_act_fn": "gelu_new",
+    "dropout_rate": 0.1,
+    "eos_token_id": 1,
+    "feed_forward_proj": "gated-gelu",
+    "initializer_factor": 1.0,
+    "is_encoder_decoder": False,
+    "is_gated_act": True,
+    "layer_norm_epsilon": 1e-06,
+    # "model_type": "t5",
+    "n_positions": 512,
+    "num_decoder_layers": 24,
+    "num_heads": 16,
+    "num_layers": 24,
+    "output_past": True,
+    "pad_token_id": 0,
+    "relative_attention_max_distance": 128,
+    "relative_attention_num_buckets": 32,
+    "tie_word_embeddings": False,
+    # "transformers_version": "4.33.3",
+    "use_cache": False,
+    "vocab_size": 32128
+}
+
+
+class MomentConfig(PretrainedConfig):
+    model_type = "moment"
+
+    def __init__(
+        self,
+        t5_config: dict = DEFAULT_T5_CONFIG,
+        d_model: int = None,
+        seq_len: int = 512,
+        patch_len: int = 16,
+        patch_stride_len: int = 16,
+        dropout: float = 0.1,
+        revin_num_features: int = 1,
+        revin_eps: float = 1e-5,
+        revin_affine: bool = True,
+        add_positional_embedding: bool = True,
+        value_embedding_bias: bool = False,
+        orth_gain: float = 1.41,
+        mask_ratio: float = 0.15,
+        freeze_embedder: bool = True,
+        freeze_encoder: bool = True,
+        freeze_head: bool = False,
+        enable_gradient_checkpointing: bool = True,
+        randomly_initialize_backbone: bool = False,
+        **kwargs
+    ):
+        self.t5_config = self._init_t5_config(t5_config)
+        self.d_model = d_model
+        self.seq_len = seq_len
+        self.patch_len = patch_len
+        self.patch_stride_len = patch_stride_len
+        self.dropout = dropout
+        self.revin_num_features = revin_num_features
+        self.revin_eps = revin_eps
+        self.revin_affine = revin_affine
+        self.add_positional_embedding = add_positional_embedding
+        self.value_embedding_bias = value_embedding_bias
+        self.orth_gain = orth_gain
+        self.mask_ratio = mask_ratio
+        self.freeze_embedder = freeze_embedder
+        self.freeze_encoder = freeze_encoder
+        self.freeze_head = freeze_head
+        self.enable_gradient_checkpointing = enable_gradient_checkpointing
+        self.randomly_initialize_backbone = randomly_initialize_backbone
+
+        self._validation_config()
+
+        super().__init__(**kwargs)
+
+    def _init_t5_config(self, config: dict):
+        if config is None:
+            return DEFAULT_T5_CONFIG
+        else:
+            # 与えられたconfigでDEFAULT_T5_CONFIGを更新
+            updated_config = DEFAULT_T5_CONFIG.copy()
+            updated_config.update(config)
+            return updated_config
+        
+    def _validation_config(self):
+        """
+        Validate configuration.
+        """
+        if self.d_model is None:
+            self.d_model = self.t5_config["d_model"]

generation_config.json

@@ -0,0 +1,7 @@
+{
+  "_from_model_config": true,
+  "bos_token_id": 1,
+  "eos_token_id": 2,
+  "pad_token_id": 32769,
+  "transformers_version": "4.41.2"
+}

model-00001-of-00007.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:8f88677cb60be3cb89350315abb985a642bee90396299aa3d3f82ef772a11147
+size 4792435960

model-00002-of-00007.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:8a3ad6610a957266f58adc61346050dc674ae55fc05a8867ac849dbf38755f50
+size 4832008160

model-00003-of-00007.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:9f10483b67b6c3e56cee90466407722f76321756c46febe5c3ad8c8cc4e3b527
+size 4999813904

model-00004-of-00007.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:3afbdc43e06689904c6cc49ba6c4a3a29e9f4d92a05d830155c6029b67edea94
+size 4999813920

model-00005-of-00007.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:3da9ed77da78e03ef944b56ebb4739ad5b81ba6c0fe75ef442a047926a5cd134
+size 4832008200

model-00006-of-00007.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:f7124c572aefe9e4c91da380eab0cae51ec510d01149a3abbbad12d8e7e981cb
+size 4999813920

model-00007-of-00007.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:13a24dcca20fe1ed97f769c727572f66b82d719331a5b7dcaf7a4a3ab1392719
+size 1007731448

model.safetensors.index.json

@@ -0,0 +1,525 @@
+{
+  "metadata": {
+    "total_size": 30463555584
+  },
+  "weight_map": {
+    "language_model.lm_head.weight": "model-00007-of-00007.safetensors",
+    "language_model.model.embed_tokens.weight": "model-00001-of-00007.safetensors",
+    "language_model.model.layers.0.input_layernorm.weight": "model-00001-of-00007.safetensors",
+    "language_model.model.layers.0.mlp.down_proj.weight": "model-00001-of-00007.safetensors",
+    "language_model.model.layers.0.mlp.gate_proj.weight": "model-00001-of-00007.safetensors",
+    "language_model.model.layers.0.mlp.up_proj.weight": "model-00001-of-00007.safetensors",
+    "language_model.model.layers.0.post_attention_layernorm.weight": "model-00001-of-00007.safetensors",
+    "language_model.model.layers.0.self_attn.k_proj.weight": "model-00001-of-00007.safetensors",
+    "language_model.model.layers.0.self_attn.o_proj.weight": "model-00001-of-00007.safetensors",
+    "language_model.model.layers.0.self_attn.q_proj.weight": "model-00001-of-00007.safetensors",
+    "language_model.model.layers.0.self_attn.v_proj.weight": "model-00001-of-00007.safetensors",
+    "language_model.model.layers.1.input_layernorm.weight": "model-00001-of-00007.safetensors",
+    "language_model.model.layers.1.mlp.down_proj.weight": "model-00001-of-00007.safetensors",
+    "language_model.model.layers.1.mlp.gate_proj.weight": "model-00001-of-00007.safetensors",
+    "language_model.model.layers.1.mlp.up_proj.weight": "model-00001-of-00007.safetensors",
+    "language_model.model.layers.1.post_attention_layernorm.weight": "model-00001-of-00007.safetensors",
+    "language_model.model.layers.1.self_attn.k_proj.weight": "model-00001-of-00007.safetensors",
+    "language_model.model.layers.1.self_attn.o_proj.weight": "model-00001-of-00007.safetensors",
+    "language_model.model.layers.1.self_attn.q_proj.weight": "model-00001-of-00007.safetensors",
+    "language_model.model.layers.1.self_attn.v_proj.weight": "model-00001-of-00007.safetensors",
+    "language_model.model.layers.10.input_layernorm.weight": "model-00003-of-00007.safetensors",
+    "language_model.model.layers.10.mlp.down_proj.weight": "model-00003-of-00007.safetensors",
+    "language_model.model.layers.10.mlp.gate_proj.weight": "model-00003-of-00007.safetensors",
+    "language_model.model.layers.10.mlp.up_proj.weight": "model-00003-of-00007.safetensors",
+    "language_model.model.layers.10.post_attention_layernorm.weight": "model-00003-of-00007.safetensors",
+    "language_model.model.layers.10.self_attn.k_proj.weight": "model-00003-of-00007.safetensors",
+    "language_model.model.layers.10.self_attn.o_proj.weight": "model-00003-of-00007.safetensors",
+    "language_model.model.layers.10.self_attn.q_proj.weight": "model-00003-of-00007.safetensors",
+    "language_model.model.layers.10.self_attn.v_proj.weight": "model-00003-of-00007.safetensors",
+    "language_model.model.layers.11.input_layernorm.weight": "model-00003-of-00007.safetensors",
+    "language_model.model.layers.11.mlp.down_proj.weight": "model-00003-of-00007.safetensors",
+    "language_model.model.layers.11.mlp.gate_proj.weight": "model-00003-of-00007.safetensors",
+    "language_model.model.layers.11.mlp.up_proj.weight": "model-00003-of-00007.safetensors",
+    "language_model.model.layers.11.post_attention_layernorm.weight": "model-00003-of-00007.safetensors",
+    "language_model.model.layers.11.self_attn.k_proj.weight": "model-00003-of-00007.safetensors",
+    "language_model.model.layers.11.self_attn.o_proj.weight": "model-00003-of-00007.safetensors",
+    "language_model.model.layers.11.self_attn.q_proj.weight": "model-00003-of-00007.safetensors",
+    "language_model.model.layers.11.self_attn.v_proj.weight": "model-00003-of-00007.safetensors",
+    "language_model.model.layers.12.input_layernorm.weight": "model-00003-of-00007.safetensors",
+    "language_model.model.layers.12.mlp.down_proj.weight": "model-00003-of-00007.safetensors",
+    "language_model.model.layers.12.mlp.gate_proj.weight": "model-00003-of-00007.safetensors",
+    "language_model.model.layers.12.mlp.up_proj.weight": "model-00003-of-00007.safetensors",
+    "language_model.model.layers.12.post_attention_layernorm.weight": "model-00003-of-00007.safetensors",
+    "language_model.model.layers.12.self_attn.k_proj.weight": "model-00003-of-00007.safetensors",
+    "language_model.model.layers.12.self_attn.o_proj.weight": "model-00003-of-00007.safetensors",
+    "language_model.model.layers.12.self_attn.q_proj.weight": "model-00003-of-00007.safetensors",
+    "language_model.model.layers.12.self_attn.v_proj.weight": "model-00003-of-00007.safetensors",
+    "language_model.model.layers.13.input_layernorm.weight": "model-00003-of-00007.safetensors",
+    "language_model.model.layers.13.mlp.down_proj.weight": "model-00003-of-00007.safetensors",
+    "language_model.model.layers.13.mlp.gate_proj.weight": "model-00003-of-00007.safetensors",
+    "language_model.model.layers.13.mlp.up_proj.weight": "model-00003-of-00007.safetensors",
+    "language_model.model.layers.13.post_attention_layernorm.weight": "model-00003-of-00007.safetensors",
+    "language_model.model.layers.13.self_attn.k_proj.weight": "model-00003-of-00007.safetensors",
+    "language_model.model.layers.13.self_attn.o_proj.weight": "model-00003-of-00007.safetensors",
+    "language_model.model.layers.13.self_attn.q_proj.weight": "model-00003-of-00007.safetensors",
+    "language_model.model.layers.13.self_attn.v_proj.weight": "model-00003-of-00007.safetensors",
+    "language_model.model.layers.14.input_layernorm.weight": "model-00004-of-00007.safetensors",
+    "language_model.model.layers.14.mlp.down_proj.weight": "model-00004-of-00007.safetensors",
+    "language_model.model.layers.14.mlp.gate_proj.weight": "model-00003-of-00007.safetensors",
+    "language_model.model.layers.14.mlp.up_proj.weight": "model-00004-of-00007.safetensors",
+    "language_model.model.layers.14.post_attention_layernorm.weight": "model-00004-of-00007.safetensors",
+    "language_model.model.layers.14.self_attn.k_proj.weight": "model-00003-of-00007.safetensors",
+    "language_model.model.layers.14.self_attn.o_proj.weight": "model-00003-of-00007.safetensors",
+    "language_model.model.layers.14.self_attn.q_proj.weight": "model-00003-of-00007.safetensors",
+    "language_model.model.layers.14.self_attn.v_proj.weight": "model-00003-of-00007.safetensors",
+    "language_model.model.layers.15.input_layernorm.weight": "model-00004-of-00007.safetensors",
+    "language_model.model.layers.15.mlp.down_proj.weight": "model-00004-of-00007.safetensors",
+    "language_model.model.layers.15.mlp.gate_proj.weight": "model-00004-of-00007.safetensors",
+    "language_model.model.layers.15.mlp.up_proj.weight": "model-00004-of-00007.safetensors",
+    "language_model.model.layers.15.post_attention_layernorm.weight": "model-00004-of-00007.safetensors",
+    "language_model.model.layers.15.self_attn.k_proj.weight": "model-00004-of-00007.safetensors",
+    "language_model.model.layers.15.self_attn.o_proj.weight": "model-00004-of-00007.safetensors",
+    "language_model.model.layers.15.self_attn.q_proj.weight": "model-00004-of-00007.safetensors",
+    "language_model.model.layers.15.self_attn.v_proj.weight": "model-00004-of-00007.safetensors",
+    "language_model.model.layers.16.input_layernorm.weight": "model-00004-of-00007.safetensors",
+    "language_model.model.layers.16.mlp.down_proj.weight": "model-00004-of-00007.safetensors",
+    "language_model.model.layers.16.mlp.gate_proj.weight": "model-00004-of-00007.safetensors",
+    "language_model.model.layers.16.mlp.up_proj.weight": "model-00004-of-00007.safetensors",
+    "language_model.model.layers.16.post_attention_layernorm.weight": "model-00004-of-00007.safetensors",
+    "language_model.model.layers.16.self_attn.k_proj.weight": "model-00004-of-00007.safetensors",
+    "language_model.model.layers.16.self_attn.o_proj.weight": "model-00004-of-00007.safetensors",
+    "language_model.model.layers.16.self_attn.q_proj.weight": "model-00004-of-00007.safetensors",
+    "language_model.model.layers.16.self_attn.v_proj.weight": "model-00004-of-00007.safetensors",
+    "language_model.model.layers.17.input_layernorm.weight": "model-00004-of-00007.safetensors",
+    "language_model.model.layers.17.mlp.down_proj.weight": "model-00004-of-00007.safetensors",
+    "language_model.model.layers.17.mlp.gate_proj.weight": "model-00004-of-00007.safetensors",
+    "language_model.model.layers.17.mlp.up_proj.weight": "model-00004-of-00007.safetensors",
+    "language_model.model.layers.17.post_attention_layernorm.weight": "model-00004-of-00007.safetensors",
+    "language_model.model.layers.17.self_attn.k_proj.weight": "model-00004-of-00007.safetensors",
+    "language_model.model.layers.17.self_attn.o_proj.weight": "model-00004-of-00007.safetensors",
+    "language_model.model.layers.17.self_attn.q_proj.weight": "model-00004-of-00007.safetensors",
+    "language_model.model.layers.17.self_attn.v_proj.weight": "model-00004-of-00007.safetensors",
+    "language_model.model.layers.18.input_layernorm.weight": "model-00004-of-00007.safetensors",
+    "language_model.model.layers.18.mlp.down_proj.weight": "model-00004-of-00007.safetensors",
+    "language_model.model.layers.18.mlp.gate_proj.weight": "model-00004-of-00007.safetensors",
+    "language_model.model.layers.18.mlp.up_proj.weight": "model-00004-of-00007.safetensors",
+    "language_model.model.layers.18.post_attention_layernorm.weight": "model-00004-of-00007.safetensors",
+    "language_model.model.layers.18.self_attn.k_proj.weight": "model-00004-of-00007.safetensors",
+    "language_model.model.layers.18.self_attn.o_proj.weight": "model-00004-of-00007.safetensors",
+    "language_model.model.layers.18.self_attn.q_proj.weight": "model-00004-of-00007.safetensors",
+    "language_model.model.layers.18.self_attn.v_proj.weight": "model-00004-of-00007.safetensors",
+    "language_model.model.layers.19.input_layernorm.weight": "model-00004-of-00007.safetensors",
+    "language_model.model.layers.19.mlp.down_proj.weight": "model-00004-of-00007.safetensors",
+    "language_model.model.layers.19.mlp.gate_proj.weight": "model-00004-of-00007.safetensors",
+    "language_model.model.layers.19.mlp.up_proj.weight": "model-00004-of-00007.safetensors",
+    "language_model.model.layers.19.post_attention_layernorm.weight": "model-00004-of-00007.safetensors",
+    "language_model.model.layers.19.self_attn.k_proj.weight": "model-00004-of-00007.safetensors",
+    "language_model.model.layers.19.self_attn.o_proj.weight": "model-00004-of-00007.safetensors",
+    "language_model.model.layers.19.self_attn.q_proj.weight": "model-00004-of-00007.safetensors",
+    "language_model.model.layers.19.self_attn.v_proj.weight": "model-00004-of-00007.safetensors",
+    "language_model.model.layers.2.input_layernorm.weight": "model-00001-of-00007.safetensors",
+    "language_model.model.layers.2.mlp.down_proj.weight": "model-00001-of-00007.safetensors",
+    "language_model.model.layers.2.mlp.gate_proj.weight": "model-00001-of-00007.safetensors",
+    "language_model.model.layers.2.mlp.up_proj.weight": "model-00001-of-00007.safetensors",
+    "language_model.model.layers.2.post_attention_layernorm.weight": "model-00001-of-00007.safetensors",
+    "language_model.model.layers.2.self_attn.k_proj.weight": "model-00001-of-00007.safetensors",
+    "language_model.model.layers.2.self_attn.o_proj.weight": "model-00001-of-00007.safetensors",
+    "language_model.model.layers.2.self_attn.q_proj.weight": "model-00001-of-00007.safetensors",
+    "language_model.model.layers.2.self_attn.v_proj.weight": "model-00001-of-00007.safetensors",
+    "language_model.model.layers.20.input_layernorm.weight": "model-00005-of-00007.safetensors",
+    "language_model.model.layers.20.mlp.down_proj.weight": "model-00005-of-00007.safetensors",
+    "language_model.model.layers.20.mlp.gate_proj.weight": "model-00005-of-00007.safetensors",
+    "language_model.model.layers.20.mlp.up_proj.weight": "model-00005-of-00007.safetensors",
+    "language_model.model.layers.20.post_attention_layernorm.weight": "model-00005-of-00007.safetensors",
+    "language_model.model.layers.20.self_attn.k_proj.weight": "model-00004-of-00007.safetensors",
+    "language_model.model.layers.20.self_attn.o_proj.weight": "model-00004-of-00007.safetensors",
+    "language_model.model.layers.20.self_attn.q_proj.weight": "model-00004-of-00007.safetensors",
+    "language_model.model.layers.20.self_attn.v_proj.weight": "model-00004-of-00007.safetensors",
+    "language_model.model.layers.21.input_layernorm.weight": "model-00005-of-00007.safetensors",
+    "language_model.model.layers.21.mlp.down_proj.weight": "model-00005-of-00007.safetensors",
+    "language_model.model.layers.21.mlp.gate_proj.weight": "model-00005-of-00007.safetensors",
+    "language_model.model.layers.21.mlp.up_proj.weight": "model-00005-of-00007.safetensors",
+    "language_model.model.layers.21.post_attention_layernorm.weight": "model-00005-of-00007.safetensors",
+    "language_model.model.layers.21.self_attn.k_proj.weight": "model-00005-of-00007.safetensors",
+    "language_model.model.layers.21.self_attn.o_proj.weight": "model-00005-of-00007.safetensors",
+    "language_model.model.layers.21.self_attn.q_proj.weight": "model-00005-of-00007.safetensors",
+    "language_model.model.layers.21.self_attn.v_proj.weight": "model-00005-of-00007.safetensors",
+    "language_model.model.layers.22.input_layernorm.weight": "model-00005-of-00007.safetensors",
+    "language_model.model.layers.22.mlp.down_proj.weight": "model-00005-of-00007.safetensors",
+    "language_model.model.layers.22.mlp.gate_proj.weight": "model-00005-of-00007.safetensors",
+    "language_model.model.layers.22.mlp.up_proj.weight": "model-00005-of-00007.safetensors",
+    "language_model.model.layers.22.post_attention_layernorm.weight": "model-00005-of-00007.safetensors",
+    "language_model.model.layers.22.self_attn.k_proj.weight": "model-00005-of-00007.safetensors",
+    "language_model.model.layers.22.self_attn.o_proj.weight": "model-00005-of-00007.safetensors",
+    "language_model.model.layers.22.self_attn.q_proj.weight": "model-00005-of-00007.safetensors",
+    "language_model.model.layers.22.self_attn.v_proj.weight": "model-00005-of-00007.safetensors",
+    "language_model.model.layers.23.input_layernorm.weight": "model-00005-of-00007.safetensors",
+    "language_model.model.layers.23.mlp.down_proj.weight": "model-00005-of-00007.safetensors",
+    "language_model.model.layers.23.mlp.gate_proj.weight": "model-00005-of-00007.safetensors",
+    "language_model.model.layers.23.mlp.up_proj.weight": "model-00005-of-00007.safetensors",
+    "language_model.model.layers.23.post_attention_layernorm.weight": "model-00005-of-00007.safetensors",
+    "language_model.model.layers.23.self_attn.k_proj.weight": "model-00005-of-00007.safetensors",
+    "language_model.model.layers.23.self_attn.o_proj.weight": "model-00005-of-00007.safetensors",
+    "language_model.model.layers.23.self_attn.q_proj.weight": "model-00005-of-00007.safetensors",
+    "language_model.model.layers.23.self_attn.v_proj.weight": "model-00005-of-00007.safetensors",
+    "language_model.model.layers.24.input_layernorm.weight": "model-00005-of-00007.safetensors",
+    "language_model.model.layers.24.mlp.down_proj.weight": "model-00005-of-00007.safetensors",
+    "language_model.model.layers.24.mlp.gate_proj.weight": "model-00005-of-00007.safetensors",
+    "language_model.model.layers.24.mlp.up_proj.weight": "model-00005-of-00007.safetensors",
+    "language_model.model.layers.24.post_attention_layernorm.weight": "model-00005-of-00007.safetensors",
+    "language_model.model.layers.24.self_attn.k_proj.weight": "model-00005-of-00007.safetensors",
+    "language_model.model.layers.24.self_attn.o_proj.weight": "model-00005-of-00007.safetensors",
+    "language_model.model.layers.24.self_attn.q_proj.weight": "model-00005-of-00007.safetensors",
+    "language_model.model.layers.24.self_attn.v_proj.weight": "model-00005-of-00007.safetensors",
+    "language_model.model.layers.25.input_layernorm.weight": "model-00006-of-00007.safetensors",
+    "language_model.model.layers.25.mlp.down_proj.weight": "model-00006-of-00007.safetensors",
+    "language_model.model.layers.25.mlp.gate_proj.weight": "model-00005-of-00007.safetensors",
+    "language_model.model.layers.25.mlp.up_proj.weight": "model-00005-of-00007.safetensors",
+    "language_model.model.layers.25.post_attention_layernorm.weight": "model-00006-of-00007.safetensors",
+    "language_model.model.layers.25.self_attn.k_proj.weight": "model-00005-of-00007.safetensors",
+    "language_model.model.layers.25.self_attn.o_proj.weight": "model-00005-of-00007.safetensors",
+    "language_model.model.layers.25.self_attn.q_proj.weight": "model-00005-of-00007.safetensors",
+    "language_model.model.layers.25.self_attn.v_proj.weight": "model-00005-of-00007.safetensors",
+    "language_model.model.layers.26.input_layernorm.weight": "model-00006-of-00007.safetensors",
+    "language_model.model.layers.26.mlp.down_proj.weight": "model-00006-of-00007.safetensors",
+    "language_model.model.layers.26.mlp.gate_proj.weight": "model-00006-of-00007.safetensors",
+    "language_model.model.layers.26.mlp.up_proj.weight": "model-00006-of-00007.safetensors",
+    "language_model.model.layers.26.post_attention_layernorm.weight": "model-00006-of-00007.safetensors",
+    "language_model.model.layers.26.self_attn.k_proj.weight": "model-00006-of-00007.safetensors",
+    "language_model.model.layers.26.self_attn.o_proj.weight": "model-00006-of-00007.safetensors",
+    "language_model.model.layers.26.self_attn.q_proj.weight": "model-00006-of-00007.safetensors",
+    "language_model.model.layers.26.self_attn.v_proj.weight": "model-00006-of-00007.safetensors",
+    "language_model.model.layers.27.input_layernorm.weight": "model-00006-of-00007.safetensors",
+    "language_model.model.layers.27.mlp.down_proj.weight": "model-00006-of-00007.safetensors",
+    "language_model.model.layers.27.mlp.gate_proj.weight": "model-00006-of-00007.safetensors",
+    "language_model.model.layers.27.mlp.up_proj.weight": "model-00006-of-00007.safetensors",
+    "language_model.model.layers.27.post_attention_layernorm.weight": "model-00006-of-00007.safetensors",
+    "language_model.model.layers.27.self_attn.k_proj.weight": "model-00006-of-00007.safetensors",
+    "language_model.model.layers.27.self_attn.o_proj.weight": "model-00006-of-00007.safetensors",
+    "language_model.model.layers.27.self_attn.q_proj.weight": "model-00006-of-00007.safetensors",
+    "language_model.model.layers.27.self_attn.v_proj.weight": "model-00006-of-00007.safetensors",
+    "language_model.model.layers.28.input_layernorm.weight": "model-00006-of-00007.safetensors",
+    "language_model.model.layers.28.mlp.down_proj.weight": "model-00006-of-00007.safetensors",
+    "language_model.model.layers.28.mlp.gate_proj.weight": "model-00006-of-00007.safetensors",
+    "language_model.model.layers.28.mlp.up_proj.weight": "model-00006-of-00007.safetensors",
+    "language_model.model.layers.28.post_attention_layernorm.weight": "model-00006-of-00007.safetensors",
+    "language_model.model.layers.28.self_attn.k_proj.weight": "model-00006-of-00007.safetensors",
+    "language_model.model.layers.28.self_attn.o_proj.weight": "model-00006-of-00007.safetensors",
+    "language_model.model.layers.28.self_attn.q_proj.weight": "model-00006-of-00007.safetensors",
+    "language_model.model.layers.28.self_attn.v_proj.weight": "model-00006-of-00007.safetensors",
+    "language_model.model.layers.29.input_layernorm.weight": "model-00006-of-00007.safetensors",
+    "language_model.model.layers.29.mlp.down_proj.weight": "model-00006-of-00007.safetensors",
+    "language_model.model.layers.29.mlp.gate_proj.weight": "model-00006-of-00007.safetensors",
+    "language_model.model.layers.29.mlp.up_proj.weight": "model-00006-of-00007.safetensors",
+    "language_model.model.layers.29.post_attention_layernorm.weight": "model-00006-of-00007.safetensors",
+    "language_model.model.layers.29.self_attn.k_proj.weight": "model-00006-of-00007.safetensors",
+    "language_model.model.layers.29.self_attn.o_proj.weight": "model-00006-of-00007.safetensors",
+    "language_model.model.layers.29.self_attn.q_proj.weight": "model-00006-of-00007.safetensors",
+    "language_model.model.layers.29.self_attn.v_proj.weight": "model-00006-of-00007.safetensors",
+    "language_model.model.layers.3.input_layernorm.weight": "model-00002-of-00007.safetensors",
+    "language_model.model.layers.3.mlp.down_proj.weight": "model-00002-of-00007.safetensors",
+    "language_model.model.layers.3.mlp.gate_proj.weight": "model-00002-of-00007.safetensors",
+    "language_model.model.layers.3.mlp.up_proj.weight": "model-00002-of-00007.safetensors",
+    "language_model.model.layers.3.post_attention_layernorm.weight": "model-00002-of-00007.safetensors",
+    "language_model.model.layers.3.self_attn.k_proj.weight": "model-00001-of-00007.safetensors",
+    "language_model.model.layers.3.self_attn.o_proj.weight": "model-00001-of-00007.safetensors",
+    "language_model.model.layers.3.self_attn.q_proj.weight": "model-00001-of-00007.safetensors",
+    "language_model.model.layers.3.self_attn.v_proj.weight": "model-00001-of-00007.safetensors",
+    "language_model.model.layers.30.input_layernorm.weight": "model-00006-of-00007.safetensors",
+    "language_model.model.layers.30.mlp.down_proj.weight": "model-00006-of-00007.safetensors",
+    "language_model.model.layers.30.mlp.gate_proj.weight": "model-00006-of-00007.safetensors",
+    "language_model.model.layers.30.mlp.up_proj.weight": "model-00006-of-00007.safetensors",
+    "language_model.model.layers.30.post_attention_layernorm.weight": "model-00006-of-00007.safetensors",
+    "language_model.model.layers.30.self_attn.k_proj.weight": "model-00006-of-00007.safetensors",
+    "language_model.model.layers.30.self_attn.o_proj.weight": "model-00006-of-00007.safetensors",
+    "language_model.model.layers.30.self_attn.q_proj.weight": "model-00006-of-00007.safetensors",
+    "language_model.model.layers.30.self_attn.v_proj.weight": "model-00006-of-00007.safetensors",
+    "language_model.model.layers.31.input_layernorm.weight": "model-00007-of-00007.safetensors",
+    "language_model.model.layers.31.mlp.down_proj.weight": "model-00007-of-00007.safetensors",
+    "language_model.model.layers.31.mlp.gate_proj.weight": "model-00006-of-00007.safetensors",
+    "language_model.model.layers.31.mlp.up_proj.weight": "model-00007-of-00007.safetensors",
+    "language_model.model.layers.31.post_attention_layernorm.weight": "model-00007-of-00007.safetensors",
+    "language_model.model.layers.31.self_attn.k_proj.weight": "model-00006-of-00007.safetensors",
+    "language_model.model.layers.31.self_attn.o_proj.weight": "model-00006-of-00007.safetensors",
+    "language_model.model.layers.31.self_attn.q_proj.weight": "model-00006-of-00007.safetensors",
+    "language_model.model.layers.31.self_attn.v_proj.weight": "model-00006-of-00007.safetensors",
+    "language_model.model.layers.4.input_layernorm.weight": "model-00002-of-00007.safetensors",
+    "language_model.model.layers.4.mlp.down_proj.weight": "model-00002-of-00007.safetensors",
+    "language_model.model.layers.4.mlp.gate_proj.weight": "model-00002-of-00007.safetensors",
+    "language_model.model.layers.4.mlp.up_proj.weight": "model-00002-of-00007.safetensors",
+    "language_model.model.layers.4.post_attention_layernorm.weight": "model-00002-of-00007.safetensors",
+    "language_model.model.layers.4.self_attn.k_proj.weight": "model-00002-of-00007.safetensors",
+    "language_model.model.layers.4.self_attn.o_proj.weight": "model-00002-of-00007.safetensors",
+    "language_model.model.layers.4.self_attn.q_proj.weight": "model-00002-of-00007.safetensors",
+    "language_model.model.layers.4.self_attn.v_proj.weight": "model-00002-of-00007.safetensors",
+    "language_model.model.layers.5.input_layernorm.weight": "model-00002-of-00007.safetensors",
+    "language_model.model.layers.5.mlp.down_proj.weight": "model-00002-of-00007.safetensors",
+    "language_model.model.layers.5.mlp.gate_proj.weight": "model-00002-of-00007.safetensors",
+    "language_model.model.layers.5.mlp.up_proj.weight": "model-00002-of-00007.safetensors",
+    "language_model.model.layers.5.post_attention_layernorm.weight": "model-00002-of-00007.safetensors",
+    "language_model.model.layers.5.self_attn.k_proj.weight": "model-00002-of-00007.safetensors",
+    "language_model.model.layers.5.self_attn.o_proj.weight": "model-00002-of-00007.safetensors",
+    "language_model.model.layers.5.self_attn.q_proj.weight": "model-00002-of-00007.safetensors",
+    "language_model.model.layers.5.self_attn.v_proj.weight": "model-00002-of-00007.safetensors",
+    "language_model.model.layers.6.input_layernorm.weight": "model-00002-of-00007.safetensors",
+    "language_model.model.layers.6.mlp.down_proj.weight": "model-00002-of-00007.safetensors",
+    "language_model.model.layers.6.mlp.gate_proj.weight": "model-00002-of-00007.safetensors",
+    "language_model.model.layers.6.mlp.up_proj.weight": "model-00002-of-00007.safetensors",
+    "language_model.model.layers.6.post_attention_layernorm.weight": "model-00002-of-00007.safetensors",
+    "language_model.model.layers.6.self_attn.k_proj.weight": "model-00002-of-00007.safetensors",
+    "language_model.model.layers.6.self_attn.o_proj.weight": "model-00002-of-00007.safetensors",
+    "language_model.model.layers.6.self_attn.q_proj.weight": "model-00002-of-00007.safetensors",
+    "language_model.model.layers.6.self_attn.v_proj.weight": "model-00002-of-00007.safetensors",
+    "language_model.model.layers.7.input_layernorm.weight": "model-00002-of-00007.safetensors",
+    "language_model.model.layers.7.mlp.down_proj.weight": "model-00002-of-00007.safetensors",
+    "language_model.model.layers.7.mlp.gate_proj.weight": "model-00002-of-00007.safetensors",
+    "language_model.model.layers.7.mlp.up_proj.weight": "model-00002-of-00007.safetensors",
+    "language_model.model.layers.7.post_attention_layernorm.weight": "model-00002-of-00007.safetensors",
+    "language_model.model.layers.7.self_attn.k_proj.weight": "model-00002-of-00007.safetensors",
+    "language_model.model.layers.7.self_attn.o_proj.weight": "model-00002-of-00007.safetensors",
+    "language_model.model.layers.7.self_attn.q_proj.weight": "model-00002-of-00007.safetensors",
+    "language_model.model.layers.7.self_attn.v_proj.weight": "model-00002-of-00007.safetensors",
+    "language_model.model.layers.8.input_layernorm.weight": "model-00003-of-00007.safetensors",
+    "language_model.model.layers.8.mlp.down_proj.weight": "model-00003-of-00007.safetensors",
+    "language_model.model.layers.8.mlp.gate_proj.weight": "model-00002-of-00007.safetensors",
+    "language_model.model.layers.8.mlp.up_proj.weight": "model-00002-of-00007.safetensors",
+    "language_model.model.layers.8.post_attention_layernorm.weight": "model-00003-of-00007.safetensors",
+    "language_model.model.layers.8.self_attn.k_proj.weight": "model-00002-of-00007.safetensors",
+    "language_model.model.layers.8.self_attn.o_proj.weight": "model-00002-of-00007.safetensors",
+    "language_model.model.layers.8.self_attn.q_proj.weight": "model-00002-of-00007.safetensors",
+    "language_model.model.layers.8.self_attn.v_proj.weight": "model-00002-of-00007.safetensors",
+    "language_model.model.layers.9.input_layernorm.weight": "model-00003-of-00007.safetensors",
+    "language_model.model.layers.9.mlp.down_proj.weight": "model-00003-of-00007.safetensors",
+    "language_model.model.layers.9.mlp.gate_proj.weight": "model-00003-of-00007.safetensors",
+    "language_model.model.layers.9.mlp.up_proj.weight": "model-00003-of-00007.safetensors",
+    "language_model.model.layers.9.post_attention_layernorm.weight": "model-00003-of-00007.safetensors",
+    "language_model.model.layers.9.self_attn.k_proj.weight": "model-00003-of-00007.safetensors",
+    "language_model.model.layers.9.self_attn.o_proj.weight": "model-00003-of-00007.safetensors",
+    "language_model.model.layers.9.self_attn.q_proj.weight": "model-00003-of-00007.safetensors",
+    "language_model.model.layers.9.self_attn.v_proj.weight": "model-00003-of-00007.safetensors",
+    "language_model.model.norm.weight": "model-00007-of-00007.safetensors",
+    "multi_modal_projector.linear_1.bias": "model-00001-of-00007.safetensors",
+    "multi_modal_projector.linear_1.weight": "model-00001-of-00007.safetensors",
+    "multi_modal_projector.linear_2.bias": "model-00001-of-00007.safetensors",
+    "multi_modal_projector.linear_2.weight": "model-00001-of-00007.safetensors",
+    "multi_modal_projector.mask_embedding": "model-00001-of-00007.safetensors",
+    "time_series_tower.encoder.block.0.layer.0.SelfAttention.k.weight": "model-00001-of-00007.safetensors",
+    "time_series_tower.encoder.block.0.layer.0.SelfAttention.o.weight": "model-00001-of-00007.safetensors",
+    "time_series_tower.encoder.block.0.layer.0.SelfAttention.q.weight": "model-00001-of-00007.safetensors",
+    "time_series_tower.encoder.block.0.layer.0.SelfAttention.relative_attention_bias.weight": "model-00001-of-00007.safetensors",
+    "time_series_tower.encoder.block.0.layer.0.SelfAttention.v.weight": "model-00001-of-00007.safetensors",
+    "time_series_tower.encoder.block.0.layer.0.layer_norm.weight": "model-00001-of-00007.safetensors",
+    "time_series_tower.encoder.block.0.layer.1.DenseReluDense.wi_0.weight": "model-00001-of-00007.safetensors",
+    "time_series_tower.encoder.block.0.layer.1.DenseReluDense.wi_1.weight": "model-00001-of-00007.safetensors",
+    "time_series_tower.encoder.block.0.layer.1.DenseReluDense.wo.weight": "model-00001-of-00007.safetensors",
+    "time_series_tower.encoder.block.0.layer.1.layer_norm.weight": "model-00001-of-00007.safetensors",
+    "time_series_tower.encoder.block.1.layer.0.SelfAttention.k.weight": "model-00001-of-00007.safetensors",
+    "time_series_tower.encoder.block.1.layer.0.SelfAttention.o.weight": "model-00001-of-00007.safetensors",
+    "time_series_tower.encoder.block.1.layer.0.SelfAttention.q.weight": "model-00001-of-00007.safetensors",
+    "time_series_tower.encoder.block.1.layer.0.SelfAttention.v.weight": "model-00001-of-00007.safetensors",
+    "time_series_tower.encoder.block.1.layer.0.layer_norm.weight": "model-00001-of-00007.safetensors",
+    "time_series_tower.encoder.block.1.layer.1.DenseReluDense.wi_0.weight": "model-00001-of-00007.safetensors",
+    "time_series_tower.encoder.block.1.layer.1.DenseReluDense.wi_1.weight": "model-00001-of-00007.safetensors",
+    "time_series_tower.encoder.block.1.layer.1.DenseReluDense.wo.weight": "model-00001-of-00007.safetensors",
+    "time_series_tower.encoder.block.1.layer.1.layer_norm.weight": "model-00001-of-00007.safetensors",
+    "time_series_tower.encoder.block.10.layer.0.SelfAttention.k.weight": "model-00001-of-00007.safetensors",
+    "time_series_tower.encoder.block.10.layer.0.SelfAttention.o.weight": "model-00001-of-00007.safetensors",
+    "time_series_tower.encoder.block.10.layer.0.SelfAttention.q.weight": "model-00001-of-00007.safetensors",
+    "time_series_tower.encoder.block.10.layer.0.SelfAttention.v.weight": "model-00001-of-00007.safetensors",
+    "time_series_tower.encoder.block.10.layer.0.layer_norm.weight": "model-00001-of-00007.safetensors",
+    "time_series_tower.encoder.block.10.layer.1.DenseReluDense.wi_0.weight": "model-00001-of-00007.safetensors",
+    "time_series_tower.encoder.block.10.layer.1.DenseReluDense.wi_1.weight": "model-00001-of-00007.safetensors",
+    "time_series_tower.encoder.block.10.layer.1.DenseReluDense.wo.weight": "model-00001-of-00007.safetensors",
+    "time_series_tower.encoder.block.10.layer.1.layer_norm.weight": "model-00001-of-00007.safetensors",
+    "time_series_tower.encoder.block.11.layer.0.SelfAttention.k.weight": "model-00001-of-00007.safetensors",
+    "time_series_tower.encoder.block.11.layer.0.SelfAttention.o.weight": "model-00001-of-00007.safetensors",
+    "time_series_tower.encoder.block.11.layer.0.SelfAttention.q.weight": "model-00001-of-00007.safetensors",
+    "time_series_tower.encoder.block.11.layer.0.SelfAttention.v.weight": "model-00001-of-00007.safetensors",
+    "time_series_tower.encoder.block.11.layer.0.layer_norm.weight": "model-00001-of-00007.safetensors",
+    "time_series_tower.encoder.block.11.layer.1.DenseReluDense.wi_0.weight": "model-00001-of-00007.safetensors",
+    "time_series_tower.encoder.block.11.layer.1.DenseReluDense.wi_1.weight": "model-00001-of-00007.safetensors",
+    "time_series_tower.encoder.block.11.layer.1.DenseReluDense.wo.weight": "model-00001-of-00007.safetensors",
+    "time_series_tower.encoder.block.11.layer.1.layer_norm.weight": "model-00001-of-00007.safetensors",
+    "time_series_tower.encoder.block.12.layer.0.SelfAttention.k.weight": "model-00001-of-00007.safetensors",
+    "time_series_tower.encoder.block.12.layer.0.SelfAttention.o.weight": "model-00001-of-00007.safetensors",
+    "time_series_tower.encoder.block.12.layer.0.SelfAttention.q.weight": "model-00001-of-00007.safetensors",
+    "time_series_tower.encoder.block.12.layer.0.SelfAttention.v.weight": "model-00001-of-00007.safetensors",
+    "time_series_tower.encoder.block.12.layer.0.layer_norm.weight": "model-00001-of-00007.safetensors",
+    "time_series_tower.encoder.block.12.layer.1.DenseReluDense.wi_0.weight": "model-00001-of-00007.safetensors",
+    "time_series_tower.encoder.block.12.layer.1.DenseReluDense.wi_1.weight": "model-00001-of-00007.safetensors",
+    "time_series_tower.encoder.block.12.layer.1.DenseReluDense.wo.weight": "model-00001-of-00007.safetensors",
+    "time_series_tower.encoder.block.12.layer.1.layer_norm.weight": "model-00001-of-00007.safetensors",
+    "time_series_tower.encoder.block.13.layer.0.SelfAttention.k.weight": "model-00001-of-00007.safetensors",
+    "time_series_tower.encoder.block.13.layer.0.SelfAttention.o.weight": "model-00001-of-00007.safetensors",
+    "time_series_tower.encoder.block.13.layer.0.SelfAttention.q.weight": "model-00001-of-00007.safetensors",
+    "time_series_tower.encoder.block.13.layer.0.SelfAttention.v.weight": "model-00001-of-00007.safetensors",
+    "time_series_tower.encoder.block.13.layer.0.layer_norm.weight": "model-00001-of-00007.safetensors",
+    "time_series_tower.encoder.block.13.layer.1.DenseReluDense.wi_0.weight": "model-00001-of-00007.safetensors",
+    "time_series_tower.encoder.block.13.layer.1.DenseReluDense.wi_1.weight": "model-00001-of-00007.safetensors",
+    "time_series_tower.encoder.block.13.layer.1.DenseReluDense.wo.weight": "model-00001-of-00007.safetensors",
+    "time_series_tower.encoder.block.13.layer.1.layer_norm.weight": "model-00001-of-00007.safetensors",
+    "time_series_tower.encoder.block.14.layer.0.SelfAttention.k.weight": "model-00001-of-00007.safetensors",
+    "time_series_tower.encoder.block.14.layer.0.SelfAttention.o.weight": "model-00001-of-00007.safetensors",
+    "time_series_tower.encoder.block.14.layer.0.SelfAttention.q.weight": "model-00001-of-00007.safetensors",
+    "time_series_tower.encoder.block.14.layer.0.SelfAttention.v.weight": "model-00001-of-00007.safetensors",
+    "time_series_tower.encoder.block.14.layer.0.layer_norm.weight": "model-00001-of-00007.safetensors",
+    "time_series_tower.encoder.block.14.layer.1.DenseReluDense.wi_0.weight": "model-00001-of-00007.safetensors",
+    "time_series_tower.encoder.block.14.layer.1.DenseReluDense.wi_1.weight": "model-00001-of-00007.safetensors",
+    "time_series_tower.encoder.block.14.layer.1.DenseReluDense.wo.weight": "model-00001-of-00007.safetensors",
+    "time_series_tower.encoder.block.14.layer.1.layer_norm.weight": "model-00001-of-00007.safetensors",
+    "time_series_tower.encoder.block.15.layer.0.SelfAttention.k.weight": "model-00001-of-00007.safetensors",
+    "time_series_tower.encoder.block.15.layer.0.SelfAttention.o.weight": "model-00001-of-00007.safetensors",
+    "time_series_tower.encoder.block.15.layer.0.SelfAttention.q.weight": "model-00001-of-00007.safetensors",
+    "time_series_tower.encoder.block.15.layer.0.SelfAttention.v.weight": "model-00001-of-00007.safetensors",
+    "time_series_tower.encoder.block.15.layer.0.layer_norm.weight": "model-00001-of-00007.safetensors",
+    "time_series_tower.encoder.block.15.layer.1.DenseReluDense.wi_0.weight": "model-00001-of-00007.safetensors",
+    "time_series_tower.encoder.block.15.layer.1.DenseReluDense.wi_1.weight": "model-00001-of-00007.safetensors",
+    "time_series_tower.encoder.block.15.layer.1.DenseReluDense.wo.weight": "model-00001-of-00007.safetensors",
+    "time_series_tower.encoder.block.15.layer.1.layer_norm.weight": "model-00001-of-00007.safetensors",
+    "time_series_tower.encoder.block.16.layer.0.SelfAttention.k.weight": "model-00001-of-00007.safetensors",
+    "time_series_tower.encoder.block.16.layer.0.SelfAttention.o.weight": "model-00001-of-00007.safetensors",
+    "time_series_tower.encoder.block.16.layer.0.SelfAttention.q.weight": "model-00001-of-00007.safetensors",
+    "time_series_tower.encoder.block.16.layer.0.SelfAttention.v.weight": "model-00001-of-00007.safetensors",
+    "time_series_tower.encoder.block.16.layer.0.layer_norm.weight": "model-00001-of-00007.safetensors",
+    "time_series_tower.encoder.block.16.layer.1.DenseReluDense.wi_0.weight": "model-00001-of-00007.safetensors",
+    "time_series_tower.encoder.block.16.layer.1.DenseReluDense.wi_1.weight": "model-00001-of-00007.safetensors",
+    "time_series_tower.encoder.block.16.layer.1.DenseReluDense.wo.weight": "model-00001-of-00007.safetensors",
+    "time_series_tower.encoder.block.16.layer.1.layer_norm.weight": "model-00001-of-00007.safetensors",
+    "time_series_tower.encoder.block.17.layer.0.SelfAttention.k.weight": "model-00001-of-00007.safetensors",
+    "time_series_tower.encoder.block.17.layer.0.SelfAttention.o.weight": "model-00001-of-00007.safetensors",
+    "time_series_tower.encoder.block.17.layer.0.SelfAttention.q.weight": "model-00001-of-00007.safetensors",
+    "time_series_tower.encoder.block.17.layer.0.SelfAttention.v.weight": "model-00001-of-00007.safetensors",
+    "time_series_tower.encoder.block.17.layer.0.layer_norm.weight": "model-00001-of-00007.safetensors",
+    "time_series_tower.encoder.block.17.layer.1.DenseReluDense.wi_0.weight": "model-00001-of-00007.safetensors",
+    "time_series_tower.encoder.block.17.layer.1.DenseReluDense.wi_1.weight": "model-00001-of-00007.safetensors",
+    "time_series_tower.encoder.block.17.layer.1.DenseReluDense.wo.weight": "model-00001-of-00007.safetensors",
+    "time_series_tower.encoder.block.17.layer.1.layer_norm.weight": "model-00001-of-00007.safetensors",
+    "time_series_tower.encoder.block.18.layer.0.SelfAttention.k.weight": "model-00001-of-00007.safetensors",
+    "time_series_tower.encoder.block.18.layer.0.SelfAttention.o.weight": "model-00001-of-00007.safetensors",
+    "time_series_tower.encoder.block.18.layer.0.SelfAttention.q.weight": "model-00001-of-00007.safetensors",
+    "time_series_tower.encoder.block.18.layer.0.SelfAttention.v.weight": "model-00001-of-00007.safetensors",
+    "time_series_tower.encoder.block.18.layer.0.layer_norm.weight": "model-00001-of-00007.safetensors",
+    "time_series_tower.encoder.block.18.layer.1.DenseReluDense.wi_0.weight": "model-00001-of-00007.safetensors",
+    "time_series_tower.encoder.block.18.layer.1.DenseReluDense.wi_1.weight": "model-00001-of-00007.safetensors",
+    "time_series_tower.encoder.block.18.layer.1.DenseReluDense.wo.weight": "model-00001-of-00007.safetensors",
+    "time_series_tower.encoder.block.18.layer.1.layer_norm.weight": "model-00001-of-00007.safetensors",
+    "time_series_tower.encoder.block.19.layer.0.SelfAttention.k.weight": "model-00001-of-00007.safetensors",
+    "time_series_tower.encoder.block.19.layer.0.SelfAttention.o.weight": "model-00001-of-00007.safetensors",
+    "time_series_tower.encoder.block.19.layer.0.SelfAttention.q.weight": "model-00001-of-00007.safetensors",
+    "time_series_tower.encoder.block.19.layer.0.SelfAttention.v.weight": "model-00001-of-00007.safetensors",
+    "time_series_tower.encoder.block.19.layer.0.layer_norm.weight": "model-00001-of-00007.safetensors",
+    "time_series_tower.encoder.block.19.layer.1.DenseReluDense.wi_0.weight": "model-00001-of-00007.safetensors",
+    "time_series_tower.encoder.block.19.layer.1.DenseReluDense.wi_1.weight": "model-00001-of-00007.safetensors",
+    "time_series_tower.encoder.block.19.layer.1.DenseReluDense.wo.weight": "model-00001-of-00007.safetensors",
+    "time_series_tower.encoder.block.19.layer.1.layer_norm.weight": "model-00001-of-00007.safetensors",
+    "time_series_tower.encoder.block.2.layer.0.SelfAttention.k.weight": "model-00001-of-00007.safetensors",
+    "time_series_tower.encoder.block.2.layer.0.SelfAttention.o.weight": "model-00001-of-00007.safetensors",
+    "time_series_tower.encoder.block.2.layer.0.SelfAttention.q.weight": "model-00001-of-00007.safetensors",
+    "time_series_tower.encoder.block.2.layer.0.SelfAttention.v.weight": "model-00001-of-00007.safetensors",
+    "time_series_tower.encoder.block.2.layer.0.layer_norm.weight": "model-00001-of-00007.safetensors",
+    "time_series_tower.encoder.block.2.layer.1.DenseReluDense.wi_0.weight": "model-00001-of-00007.safetensors",
+    "time_series_tower.encoder.block.2.layer.1.DenseReluDense.wi_1.weight": "model-00001-of-00007.safetensors",
+    "time_series_tower.encoder.block.2.layer.1.DenseReluDense.wo.weight": "model-00001-of-00007.safetensors",
+    "time_series_tower.encoder.block.2.layer.1.layer_norm.weight": "model-00001-of-00007.safetensors",
+    "time_series_tower.encoder.block.20.layer.0.SelfAttention.k.weight": "model-00001-of-00007.safetensors",
+    "time_series_tower.encoder.block.20.layer.0.SelfAttention.o.weight": "model-00001-of-00007.safetensors",
+    "time_series_tower.encoder.block.20.layer.0.SelfAttention.q.weight": "model-00001-of-00007.safetensors",
+    "time_series_tower.encoder.block.20.layer.0.SelfAttention.v.weight": "model-00001-of-00007.safetensors",
+    "time_series_tower.encoder.block.20.layer.0.layer_norm.weight": "model-00001-of-00007.safetensors",
+    "time_series_tower.encoder.block.20.layer.1.DenseReluDense.wi_0.weight": "model-00001-of-00007.safetensors",
+    "time_series_tower.encoder.block.20.layer.1.DenseReluDense.wi_1.weight": "model-00001-of-00007.safetensors",
+    "time_series_tower.encoder.block.20.layer.1.DenseReluDense.wo.weight": "model-00001-of-00007.safetensors",
+    "time_series_tower.encoder.block.20.layer.1.layer_norm.weight": "model-00001-of-00007.safetensors",
+    "time_series_tower.encoder.block.21.layer.0.SelfAttention.k.weight": "model-00001-of-00007.safetensors",
+    "time_series_tower.encoder.block.21.layer.0.SelfAttention.o.weight": "model-00001-of-00007.safetensors",
+    "time_series_tower.encoder.block.21.layer.0.SelfAttention.q.weight": "model-00001-of-00007.safetensors",
+    "time_series_tower.encoder.block.21.layer.0.SelfAttention.v.weight": "model-00001-of-00007.safetensors",
+    "time_series_tower.encoder.block.21.layer.0.layer_norm.weight": "model-00001-of-00007.safetensors",
+    "time_series_tower.encoder.block.21.layer.1.DenseReluDense.wi_0.weight": "model-00001-of-00007.safetensors",
+    "time_series_tower.encoder.block.21.layer.1.DenseReluDense.wi_1.weight": "model-00001-of-00007.safetensors",
+    "time_series_tower.encoder.block.21.layer.1.DenseReluDense.wo.weight": "model-00001-of-00007.safetensors",
+    "time_series_tower.encoder.block.21.layer.1.layer_norm.weight": "model-00001-of-00007.safetensors",
+    "time_series_tower.encoder.block.22.layer.0.SelfAttention.k.weight": "model-00001-of-00007.safetensors",
+    "time_series_tower.encoder.block.22.layer.0.SelfAttention.o.weight": "model-00001-of-00007.safetensors",
+    "time_series_tower.encoder.block.22.layer.0.SelfAttention.q.weight": "model-00001-of-00007.safetensors",
+    "time_series_tower.encoder.block.22.layer.0.SelfAttention.v.weight": "model-00001-of-00007.safetensors",
+    "time_series_tower.encoder.block.22.layer.0.layer_norm.weight": "model-00001-of-00007.safetensors",
+    "time_series_tower.encoder.block.22.layer.1.DenseReluDense.wi_0.weight": "model-00001-of-00007.safetensors",
+    "time_series_tower.encoder.block.22.layer.1.DenseReluDense.wi_1.weight": "model-00001-of-00007.safetensors",
+    "time_series_tower.encoder.block.22.layer.1.DenseReluDense.wo.weight": "model-00001-of-00007.safetensors",
+    "time_series_tower.encoder.block.22.layer.1.layer_norm.weight": "model-00001-of-00007.safetensors",
+    "time_series_tower.encoder.block.23.layer.0.SelfAttention.k.weight": "model-00001-of-00007.safetensors",
+    "time_series_tower.encoder.block.23.layer.0.SelfAttention.o.weight": "model-00001-of-00007.safetensors",
+    "time_series_tower.encoder.block.23.layer.0.SelfAttention.q.weight": "model-00001-of-00007.safetensors",
+    "time_series_tower.encoder.block.23.layer.0.SelfAttention.v.weight": "model-00001-of-00007.safetensors",
+    "time_series_tower.encoder.block.23.layer.0.layer_norm.weight": "model-00001-of-00007.safetensors",
+    "time_series_tower.encoder.block.23.layer.1.DenseReluDense.wi_0.weight": "model-00001-of-00007.safetensors",
+    "time_series_tower.encoder.block.23.layer.1.DenseReluDense.wi_1.weight": "model-00001-of-00007.safetensors",
+    "time_series_tower.encoder.block.23.layer.1.DenseReluDense.wo.weight": "model-00001-of-00007.safetensors",
+    "time_series_tower.encoder.block.23.layer.1.layer_norm.weight": "model-00001-of-00007.safetensors",
+    "time_series_tower.encoder.block.3.layer.0.SelfAttention.k.weight": "model-00001-of-00007.safetensors",
+    "time_series_tower.encoder.block.3.layer.0.SelfAttention.o.weight": "model-00001-of-00007.safetensors",
+    "time_series_tower.encoder.block.3.layer.0.SelfAttention.q.weight": "model-00001-of-00007.safetensors",
+    "time_series_tower.encoder.block.3.layer.0.SelfAttention.v.weight": "model-00001-of-00007.safetensors",
+    "time_series_tower.encoder.block.3.layer.0.layer_norm.weight": "model-00001-of-00007.safetensors",
+    "time_series_tower.encoder.block.3.layer.1.DenseReluDense.wi_0.weight": "model-00001-of-00007.safetensors",
+    "time_series_tower.encoder.block.3.layer.1.DenseReluDense.wi_1.weight": "model-00001-of-00007.safetensors",
+    "time_series_tower.encoder.block.3.layer.1.DenseReluDense.wo.weight": "model-00001-of-00007.safetensors",
+    "time_series_tower.encoder.block.3.layer.1.layer_norm.weight": "model-00001-of-00007.safetensors",
+    "time_series_tower.encoder.block.4.layer.0.SelfAttention.k.weight": "model-00001-of-00007.safetensors",
+    "time_series_tower.encoder.block.4.layer.0.SelfAttention.o.weight": "model-00001-of-00007.safetensors",
+    "time_series_tower.encoder.block.4.layer.0.SelfAttention.q.weight": "model-00001-of-00007.safetensors",
+    "time_series_tower.encoder.block.4.layer.0.SelfAttention.v.weight": "model-00001-of-00007.safetensors",
+    "time_series_tower.encoder.block.4.layer.0.layer_norm.weight": "model-00001-of-00007.safetensors",
+    "time_series_tower.encoder.block.4.layer.1.DenseReluDense.wi_0.weight": "model-00001-of-00007.safetensors",
+    "time_series_tower.encoder.block.4.layer.1.DenseReluDense.wi_1.weight": "model-00001-of-00007.safetensors",
+    "time_series_tower.encoder.block.4.layer.1.DenseReluDense.wo.weight": "model-00001-of-00007.safetensors",
+    "time_series_tower.encoder.block.4.layer.1.layer_norm.weight": "model-00001-of-00007.safetensors",
+    "time_series_tower.encoder.block.5.layer.0.SelfAttention.k.weight": "model-00001-of-00007.safetensors",
+    "time_series_tower.encoder.block.5.layer.0.SelfAttention.o.weight": "model-00001-of-00007.safetensors",
+    "time_series_tower.encoder.block.5.layer.0.SelfAttention.q.weight": "model-00001-of-00007.safetensors",
+    "time_series_tower.encoder.block.5.layer.0.SelfAttention.v.weight": "model-00001-of-00007.safetensors",
+    "time_series_tower.encoder.block.5.layer.0.layer_norm.weight": "model-00001-of-00007.safetensors",
+    "time_series_tower.encoder.block.5.layer.1.DenseReluDense.wi_0.weight": "model-00001-of-00007.safetensors",
+    "time_series_tower.encoder.block.5.layer.1.DenseReluDense.wi_1.weight": "model-00001-of-00007.safetensors",
+    "time_series_tower.encoder.block.5.layer.1.DenseReluDense.wo.weight": "model-00001-of-00007.safetensors",
+    "time_series_tower.encoder.block.5.layer.1.layer_norm.weight": "model-00001-of-00007.safetensors",
+    "time_series_tower.encoder.block.6.layer.0.SelfAttention.k.weight": "model-00001-of-00007.safetensors",
+    "time_series_tower.encoder.block.6.layer.0.SelfAttention.o.weight": "model-00001-of-00007.safetensors",
+    "time_series_tower.encoder.block.6.layer.0.SelfAttention.q.weight": "model-00001-of-00007.safetensors",
+    "time_series_tower.encoder.block.6.layer.0.SelfAttention.v.weight": "model-00001-of-00007.safetensors",
+    "time_series_tower.encoder.block.6.layer.0.layer_norm.weight": "model-00001-of-00007.safetensors",
+    "time_series_tower.encoder.block.6.layer.1.DenseReluDense.wi_0.weight": "model-00001-of-00007.safetensors",
+    "time_series_tower.encoder.block.6.layer.1.DenseReluDense.wi_1.weight": "model-00001-of-00007.safetensors",
+    "time_series_tower.encoder.block.6.layer.1.DenseReluDense.wo.weight": "model-00001-of-00007.safetensors",
+    "time_series_tower.encoder.block.6.layer.1.layer_norm.weight": "model-00001-of-00007.safetensors",
+    "time_series_tower.encoder.block.7.layer.0.SelfAttention.k.weight": "model-00001-of-00007.safetensors",
+    "time_series_tower.encoder.block.7.layer.0.SelfAttention.o.weight": "model-00001-of-00007.safetensors",
+    "time_series_tower.encoder.block.7.layer.0.SelfAttention.q.weight": "model-00001-of-00007.safetensors",
+    "time_series_tower.encoder.block.7.layer.0.SelfAttention.v.weight": "model-00001-of-00007.safetensors",
+    "time_series_tower.encoder.block.7.layer.0.layer_norm.weight": "model-00001-of-00007.safetensors",
+    "time_series_tower.encoder.block.7.layer.1.DenseReluDense.wi_0.weight": "model-00001-of-00007.safetensors",
+    "time_series_tower.encoder.block.7.layer.1.DenseReluDense.wi_1.weight": "model-00001-of-00007.safetensors",
+    "time_series_tower.encoder.block.7.layer.1.DenseReluDense.wo.weight": "model-00001-of-00007.safetensors",
+    "time_series_tower.encoder.block.7.layer.1.layer_norm.weight": "model-00001-of-00007.safetensors",
+    "time_series_tower.encoder.block.8.layer.0.SelfAttention.k.weight": "model-00001-of-00007.safetensors",
+    "time_series_tower.encoder.block.8.layer.0.SelfAttention.o.weight": "model-00001-of-00007.safetensors",
+    "time_series_tower.encoder.block.8.layer.0.SelfAttention.q.weight": "model-00001-of-00007.safetensors",
+    "time_series_tower.encoder.block.8.layer.0.SelfAttention.v.weight": "model-00001-of-00007.safetensors",
+    "time_series_tower.encoder.block.8.layer.0.layer_norm.weight": "model-00001-of-00007.safetensors",
+    "time_series_tower.encoder.block.8.layer.1.DenseReluDense.wi_0.weight": "model-00001-of-00007.safetensors",
+    "time_series_tower.encoder.block.8.layer.1.DenseReluDense.wi_1.weight": "model-00001-of-00007.safetensors",
+    "time_series_tower.encoder.block.8.layer.1.DenseReluDense.wo.weight": "model-00001-of-00007.safetensors",
+    "time_series_tower.encoder.block.8.layer.1.layer_norm.weight": "model-00001-of-00007.safetensors",
+    "time_series_tower.encoder.block.9.layer.0.SelfAttention.k.weight": "model-00001-of-00007.safetensors",
+    "time_series_tower.encoder.block.9.layer.0.SelfAttention.o.weight": "model-00001-of-00007.safetensors",
+    "time_series_tower.encoder.block.9.layer.0.SelfAttention.q.weight": "model-00001-of-00007.safetensors",
+    "time_series_tower.encoder.block.9.layer.0.SelfAttention.v.weight": "model-00001-of-00007.safetensors",
+    "time_series_tower.encoder.block.9.layer.0.layer_norm.weight": "model-00001-of-00007.safetensors",
+    "time_series_tower.encoder.block.9.layer.1.DenseReluDense.wi_0.weight": "model-00001-of-00007.safetensors",
+    "time_series_tower.encoder.block.9.layer.1.DenseReluDense.wi_1.weight": "model-00001-of-00007.safetensors",
+    "time_series_tower.encoder.block.9.layer.1.DenseReluDense.wo.weight": "model-00001-of-00007.safetensors",
+    "time_series_tower.encoder.block.9.layer.1.layer_norm.weight": "model-00001-of-00007.safetensors",
+    "time_series_tower.encoder.embed_tokens.weight": "model-00001-of-00007.safetensors",
+    "time_series_tower.encoder.final_layer_norm.weight": "model-00001-of-00007.safetensors",
+    "time_series_tower.patch_embedding.mask_embedding": "model-00001-of-00007.safetensors",
+    "time_series_tower.patch_embedding.position_embedding.pe": "model-00001-of-00007.safetensors",
+    "time_series_tower.patch_embedding.value_embedding.weight": "model-00001-of-00007.safetensors"
+  }
+}

modeling_mists.py

@@ -0,0 +1,403 @@
+from dataclasses import dataclass
+from typing import List, Optional, Tuple, Union
+
+import torch
+import torch.utils.checkpoint
+from torch import nn
+
+from transformers import PreTrainedModel
+from transformers.activations import ACT2FN
+from transformers import Cache
+from transformers.modeling_outputs import ModelOutput
+from transformers.utils import (
+    add_start_docstrings,
+    add_start_docstrings_to_model_forward,
+    logging,
+    replace_return_docstrings,
+)
+from transformers import AutoModel, AutoModelForCausalLM
+
+from .modeling_moment import MomentEmbeddingModel
+from .configuration_mists import MistsConfig
+
+
+@dataclass
+# Copied from transformers.models.idefics.modeling_idefics.IdeficsCausalLMOutputWithPast with Idefics->Mists
+class MistsCausalLMOutputWithPast(ModelOutput):
+    loss: Optional[torch.FloatTensor] = None
+    logits: torch.FloatTensor = None
+    past_key_values: Optional[List[torch.FloatTensor]] = None
+    hidden_states: Optional[Tuple[torch.FloatTensor]] = None
+    attentions: Optional[Tuple[torch.FloatTensor]] = None
+    time_series_hidden_states: Optional[Tuple[torch.FloatTensor]] = None
+
+
+class MistsMultiModalProjector(nn.Module):
+    def __init__(self, config: MistsConfig):
+        super().__init__()
+
+        # time series towerからのoutputは定型でない。input_maskに合わせてpadding用の学習可能なベクトルを使用し、time series towerからの入力を定型にする。
+        self.mask_embedding = nn.Parameter(torch.randn(1, 1, config.time_series_hidden_size))
+        
+        # mlp
+        self.linear_1 = nn.Linear(config.time_series_hidden_size, config.text_config.hidden_size, bias=True)
+        self.act = ACT2FN[config.projector_hidden_act]
+        self.linear_2 = nn.Linear(config.text_config.hidden_size, config.text_config.hidden_size, bias=True)
+
+    def forward(self, time_series_features, input_mask):
+        masked_features = time_series_features * input_mask.unsqueeze(-1) + self.mask_embedding * (1 - input_mask.unsqueeze(-1))
+        hidden_states = self.linear_1(masked_features)
+        hidden_states = self.act(hidden_states)
+        hidden_states = self.linear_2(hidden_states)
+        return hidden_states
+    
+
+class MistsPreTrainedModel(PreTrainedModel):
+    config_class = MistsConfig
+    base_model_prefix = "model"
+    supports_gradient_checkpointing = True
+    _no_split_modules = ["T5Block"]
+    _skip_keys_device_placement = "past_key_values"
+    _supports_flash_attn_2 = True
+    _supports_sdpa = True
+    _supports_cache_class = True
+    _supports_static_cache = True
+
+    def _init_weights(self, module):
+        # important: 現状Mistralの初期化コードをそのまま移植している。
+        # refers: https://github.com/huggingface/transformers/blob/25245ec26dc29bcf6102e1b4ddd0dfd02e720cf5/src/transformers/models/mistral/modeling_mistral.py#L762
+        # 現状のまま事前学習を行うのは望ましくなく、FineTuningと推論のみが可能。
+        std = self.config.text_config.initializer_range
+        if isinstance(module, nn.Linear):
+            module.weight.data.normal_(mean=0.0, std=std)
+            if module.bias is not None:
+                module.bias.data.zero_()
+        elif isinstance(module, nn.Embedding):
+            module.weight.data.normal_(mean=0.0, std=std)
+            if module.padding_idx is not None:
+                module.weight.data[module.padding_idx].zero_()
+
+
+class MistsForConditionalGeneration(MistsPreTrainedModel):
+    def __init__(self, config: MistsConfig):
+        super().__init__(config)
+
+        self.time_series_tower = MomentEmbeddingModel(config.time_series_config)
+        self.multi_modal_projector = MistsMultiModalProjector(config)
+        self.vocab_size = config.text_config.vocab_size
+        self.language_model = AutoModelForCausalLM.from_config(
+            config.text_config, attn_implementation=config._attn_implementation
+        )
+        self.pad_token_id = self.config.pad_token_id if self.config.pad_token_id is not None else -1
+        self.post_init()
+
+    def get_time_series_tower(self):
+        time_series_tower = getattr(self, 'time_series_tower', None)
+        if type(time_series_tower) is list:
+            time_series_tower = time_series_tower[0]
+        return time_series_tower
+
+    def get_input_embeddings(self):
+        return self.language_model.get_input_embeddings()
+
+    def set_input_embeddings(self, value):
+        self.language_model.set_input_embeddings(value)
+
+    def get_output_embeddings(self):
+        return self.language_model.get_output_embeddings()
+
+    def set_output_embeddings(self, new_embeddings):
+        self.language_model.set_output_embeddings(new_embeddings)
+
+    def set_decoder(self, decoder):
+        self.language_model.set_decoder(decoder)
+
+    def get_decoder(self):
+        return self.language_model.get_decoder()
+
+    def tie_weights(self):
+        return self.language_model.tie_weights()
+
+    def resize_token_embeddings(self, new_num_tokens: Optional[int] = None, pad_to_multiple_of=None) -> nn.Embedding:
+        model_embeds = self.language_model.resize_token_embeddings(new_num_tokens, pad_to_multiple_of)
+        # update vocab size
+        self.config.text_config.vocab_size = model_embeds.num_embeddings
+        self.vocab_size = model_embeds.num_embeddings
+        return model_embeds
+    
+    # copy _merge_input_ids_with_image_features from LlabaForConditionalGeneration
+    # refers: https://github.com/huggingface/transformers/blob/25245ec26dc29bcf6102e1b4ddd0dfd02e720cf5/src/transformers/models/llava/modeling_llava.py#L277C9-L277C45
+    def _merge_input_ids_with_time_series_features(self, time_series_features, inputs_embeds, input_ids, attention_mask, labels):
+        num_time_series, num_time_series_patches, embed_dim = time_series_features.shape  # num_time_series_patches = n_channels x n_patches
+        batch_size, sequence_length = input_ids.shape
+        left_padding = not torch.sum(input_ids[:, -1] == torch.tensor(self.pad_token_id))
+        # 1. Create a mask to know where special time_series tokens are
+        special_time_series_token_mask = input_ids == self.config.time_series_token_index
+        num_special_time_series_tokens = torch.sum(special_time_series_token_mask, dim=-1)
+        # Compute the maximum embed dimension
+        max_embed_dim = (num_special_time_series_tokens.max() * (num_time_series_patches - 1)) + sequence_length
+        max_embed_dim = int(max_embed_dim.item())  # テンソルから整数値を取得
+        if max_embed_dim is None:
+            print(f"num_special_time_series_tokens.max(): {num_special_time_series_tokens.max()}")
+            print(f"num_time_series_patches: {num_time_series_patches}")
+            print(f"sequence_length: {sequence_length}")
+        else:
+            print(f"max_embed_dim 0: {max_embed_dim}")
+        batch_indices, non_time_series_indices = torch.where(input_ids != self.config.time_series_token_index)
+
+        # 2. Compute the positions where text should be written
+        # Calculate new positions for text tokens in merged time_series-text sequence.
+        # `special_time_series_token_mask` identifies time_series tokens. Each time_series token will be replaced by `nb_text_tokens_per_time_series - 1` text tokens.
+        # `torch.cumsum` computes how each time_series token shifts subsequent text token positions.
+        # - 1 to adjust for zero-based indexing, as `cumsum` inherently increases indices by one.
+        new_token_positions = torch.cumsum((special_time_series_token_mask * (num_time_series_patches - 1) + 1), -1) - 1
+        nb_time_series_pad = max_embed_dim - 1 - new_token_positions[:, -1]
+        if left_padding:
+            new_token_positions += nb_time_series_pad[:, None]  # offset for left padding
+        text_to_overwrite = new_token_positions[batch_indices, non_time_series_indices]
+
+        # 3. Create the full embedding, already padded to the maximum position
+        final_embedding = torch.zeros(
+            batch_size, max_embed_dim, embed_dim, dtype=inputs_embeds.dtype, device=inputs_embeds.device
+        )
+        final_attention_mask = torch.zeros(
+            batch_size, max_embed_dim, dtype=attention_mask.dtype, device=inputs_embeds.device
+        )
+        if labels is not None:
+            final_labels = torch.full(
+                (batch_size, max_embed_dim), self.config.ignore_index, dtype=input_ids.dtype, device=input_ids.device
+            )
+        # In case the Vision model or the Language model has been offloaded to CPU, we need to manually
+        # set the corresponding tensors into their correct target device.
+        target_device = inputs_embeds.device
+        batch_indices, non_time_series_indices, text_to_overwrite = (
+            batch_indices.to(target_device),
+            non_time_series_indices.to(target_device),
+            text_to_overwrite.to(target_device),
+        )
+        attention_mask = attention_mask.to(target_device)
+
+        # 4. Fill the embeddings based on the mask. If we have ["hey" "<time_series>", "how", "are"]
+        # we need to index copy on [0, 577, 578, 579] for the text and [1:576] for the time_series features
+        final_embedding[batch_indices, text_to_overwrite] = inputs_embeds[batch_indices, non_time_series_indices]
+        final_attention_mask[batch_indices, text_to_overwrite] = attention_mask[batch_indices, non_time_series_indices]
+        print("max_embed_dim is None: ", (max_embed_dim is None))
+        print("max_embed_dim: ", max_embed_dim)
+        if labels is not None:
+            final_labels[batch_indices, text_to_overwrite] = labels[batch_indices, non_time_series_indices]
+        print("max_embed_dim is None: ", (max_embed_dim is None))
+        print("max_embed_dim: ", max_embed_dim)
+
+        # 5. Fill the embeddings corresponding to the time_series. Anything that is not `text_positions` needs filling (#29835)
+        print("inputs_embeds.device: ", inputs_embeds.device)
+        print("max_embed_dim: ", max_embed_dim, " is None: ", (max_embed_dim is None))
+        time_series_to_overwrite = torch.full(
+            (batch_size, max_embed_dim), True, dtype=torch.bool, device=inputs_embeds.device
+        )
+        time_series_to_overwrite[batch_indices, text_to_overwrite] = False
+        time_series_to_overwrite &= time_series_to_overwrite.cumsum(-1) - 1 >= nb_time_series_pad[:, None].to(target_device)
+
+        if time_series_to_overwrite.sum() != time_series_features.shape[:-1].numel():
+            raise ValueError(
+                f"The input provided to the model are wrong. The number of time series tokens is {torch.sum(special_time_series_token_mask)} while"
+                f" the number of time series given to the model is {num_time_series}. This prevents correct indexing and breaks batch generation."
+            )
+
+        final_embedding[time_series_to_overwrite] = time_series_features.contiguous().reshape(-1, embed_dim).to(target_device)
+        final_attention_mask |= time_series_to_overwrite
+        position_ids = (final_attention_mask.cumsum(-1) - 1).masked_fill_((final_attention_mask == 0), 1)
+
+        # 6. Mask out the embedding at padding positions, as we later use the past_key_value value to determine the non-attended tokens.
+        batch_indices, pad_indices = torch.where(input_ids == self.pad_token_id)
+        indices_to_mask = new_token_positions[batch_indices, pad_indices]
+
+        final_embedding[batch_indices, indices_to_mask] = 0
+
+        if labels is None:
+            final_labels = None
+
+        return final_embedding, final_attention_mask, final_labels, position_ids
+    
+    def forward(
+        self,
+        input_ids: torch.LongTensor = None,
+        time_series_values: torch.FloatTensor = None,
+        time_series_input_mask: torch.FloatTensor = None,
+        attention_mask: Optional[torch.Tensor] = None,
+        position_ids: Optional[torch.LongTensor] = None,
+        past_key_values: Optional[List[torch.FloatTensor]] = None,
+        inputs_embeds: Optional[torch.FloatTensor] = None,
+        # time_series_feature_layer: Optional[int] = None,
+        # time_series_feature_select_strategy: Optional[str] = None,
+        labels: Optional[torch.LongTensor] = None,
+        use_cache: Optional[bool] = None,
+        output_attentions: Optional[bool] = None,
+        output_hidden_states: Optional[bool] = None,
+        return_dict: Optional[bool] = None,
+    ) -> Union[Tuple, MistsCausalLMOutputWithPast]:
+        
+        # language_modelの引数で変わる
+        # output_attentions = output_attentions if output_attentions is not None else self.config.output_attentions
+        # output_hidden_states = (
+        #     output_hidden_states if output_hidden_states is not None else self.config.output_hidden_states
+        # )
+        # return_dict = return_dict if return_dict is not None else self.config.use_return_dict
+        # vision_feature_layer = (
+        #     vision_feature_layer if vision_feature_layer is not None else self.config.vision_feature_layer
+        # )
+        # vision_feature_select_strategy = (
+        #     vision_feature_select_strategy
+        #     if vision_feature_select_strategy is not None
+        #     else self.config.vision_feature_select_strategy
+        # )
+
+        if inputs_embeds is None:
+            # 1. Extra the input embeddings
+            inputs_embeds = self.get_input_embeddings()(input_ids)
+
+            # 2. Merge text and time_series
+            if time_series_values is not None and input_ids.shape[1] != 1:
+                time_series_outputs = self.time_series_tower(time_series_values, time_series_input_mask)
+                time_series_features = self.multi_modal_projector(
+                    time_series_features=time_series_outputs.hidden_states,  # [batch_size, n_patches, d_model]
+                    input_mask=time_series_outputs.input_mask_patch_view,    # [batch_size, n_paches]
+                )
+
+                inputs_embeds = inputs_embeds.to(time_series_features.dtype)
+                inputs_embeds, attention_mask, labels, position_ids =self._merge_input_ids_with_time_series_features(
+                    time_series_features, inputs_embeds, input_ids, attention_mask, labels
+                )
+
+            # In case input_ids.shape[1] == 1 & time_series_values==None & past_key_values != None, we are in the case of
+            # generation with cache
+            elif past_key_values is not None and time_series_values is not None and input_ids.shape[1] == 1:
+                # Retrieve the first layer to inspect the logits and mask out the hidden states
+                # that are set to 0
+                first_layer_past_key_value = past_key_values[0][0][:, :, :, 0]
+
+                # Sum all dimensions of head_dim (-2) to avoid random errors such as: https://github.com/huggingface/transformers/pull/28032#issuecomment-1863691941
+                batch_index, non_attended_tokens = torch.where(first_layer_past_key_value.float().sum(-2) == 0)
+
+                # Get the target length
+                target_length = input_ids.shape[1]
+                past_length = first_layer_past_key_value.shape[-1]
+
+                extended_attention_mask = torch.ones(
+                    (attention_mask.shape[0], past_length),
+                    dtype=attention_mask.dtype,
+                    device=attention_mask.device,
+                )
+
+                # Filter out only the tokens that can be un-attended, this can happen
+                # if one uses Llava + Fused modules where the cache on the
+                # first iteration is already big enough, or if one passes custom cache
+                valid_indices = non_attended_tokens < extended_attention_mask.size(-1)
+                new_batch_index = batch_index[valid_indices]
+                new_non_attended_tokens = non_attended_tokens[valid_indices]
+
+                # Zero-out the places where we don't need to attend
+                extended_attention_mask[new_batch_index, new_non_attended_tokens] = 0
+
+                attention_mask = torch.cat((extended_attention_mask, attention_mask[:, -target_length:]), dim=1)
+                position_ids = torch.sum(attention_mask, dim=1).unsqueeze(-1) - 1
+            
+        outputs = self.language_model(
+            attention_mask=attention_mask,
+            position_ids=position_ids,
+            past_key_values=past_key_values,
+            inputs_embeds=inputs_embeds.to(self.language_model.dtype),
+            use_cache=use_cache,
+            output_attentions=output_attentions,
+            output_hidden_states=output_hidden_states,
+            return_dict=return_dict,
+        )
+
+        logits = outputs[0]
+
+        loss = None
+        if labels is not None:
+            # Shift so that tokens < n predict n
+            if attention_mask is not None:
+                shift_attention_mask = attention_mask[..., 1:]
+                shift_logits = logits[..., :-1, :][shift_attention_mask.to(logits.device) != 0].contiguous()
+                shift_labels = labels[..., 1:][shift_attention_mask.to(labels.device) != 0].contiguous()
+            else:
+                shift_logits = logits[..., :-1, :].contiguous()
+                shift_labels = labels[..., 1:].contiguous()
+            # Flatten the tokens
+            loss_fct = nn.CrossEntropyLoss()
+            loss = loss_fct(
+                shift_logits.view(-1, shift_logits.size(-1)), shift_labels.view(-1).to(shift_logits.device)
+            )
+
+        if not return_dict:
+            output = (logits,) + outputs[1:]
+            return (loss,) + output if loss is not None else output
+        
+        return MistsCausalLMOutputWithPast(
+            loss=loss,
+            logits=logits,
+            past_key_values=outputs.past_key_values,
+            hidden_states=outputs.hidden_states,
+            attentions=outputs.attentions,
+        )
+    
+    def prepare_inputs_for_generation(
+        self, input_ids, past_key_values=None, inputs_embeds=None, time_series_values=None, attention_mask=None, **kwargs
+    ):
+        if past_key_values is not None:
+            if isinstance(past_key_values, Cache):
+                cache_length = past_key_values.get_seq_length()
+                past_length = past_key_values.seen_tokens
+            else:
+                cache_length = past_length = past_key_values[0][0].shape[2]
+
+            # Keep only the unprocessed tokens:
+            # 1 - If the length of the attention_mask exceeds the length of input_ids, then we are in a setting where
+            # some of the inputs are exclusively passed as part of the cache (e.g. when passing input_embeds as
+            # input)
+            if attention_mask is not None and attention_mask.shape[1] > input_ids.shape[1]:
+                input_ids = input_ids[:, -(attention_mask.shape[1] - past_length) :]
+            # 2 - If the past_length is smaller than input_ids', then input_ids holds all input tokens. We can discard
+            # input_ids based on the past_length.
+            elif past_length < input_ids.shape[1]:
+                input_ids = input_ids[:, past_length:]
+            # 3 - Otherwise (past_length >= input_ids.shape[1]), let's assume input_ids only has unprocessed tokens.
+            elif self.config.time_series_token_index in input_ids:
+                input_ids = input_ids[:, input_ids.shape[1] - 1 :]
+            # If the cache has seen more tokens than it can hold, then the cache has a size limit. Let's discard the
+            # older attention values, as their corresponding values are not part of the input.
+            if cache_length < past_length and attention_mask is not None:
+                attention_mask = attention_mask[:, -(cache_length + input_ids.shape[1]) :]
+
+        position_ids = kwargs.get("position_ids", None)
+        if attention_mask is not None and position_ids is None:
+            # create position_ids on the fly for batch generation
+            position_ids = attention_mask.long().cumsum(-1) - 1
+            position_ids.masked_fill_(attention_mask == 0, 1)
+            if past_key_values:
+                position_ids = position_ids[:, -input_ids.shape[1] :]
+
+        # if `inputs_embeds` are passed, we only want to use them in the 1st generation step
+        if inputs_embeds is not None and past_key_values is None:
+            model_inputs = {"inputs_embeds": inputs_embeds}
+        else:
+            model_inputs = {"input_ids": input_ids}
+
+        model_inputs.update(
+            {
+                "position_ids": position_ids,
+                "past_key_values": past_key_values,
+                "use_cache": kwargs.get("use_cache"),
+                "attention_mask": attention_mask,
+                "time_series_values": time_series_values,
+            }
+        )
+        return model_inputs
+    
+    def _reorder_cache(self, *args, **kwargs):
+        return self.language_model._reorder_cache(*args, **kwargs)
+
+
+
+

modeling_moment.py

@@ -0,0 +1,533 @@
+# Auton LabによるMomentライブラリをTransformers向けに書き換えたものです。
+# Embeddingに特化したアーキテクチャとなっています。
+# refers: https://github.com/moment-timeseries-foundation-model/moment
+
+from dataclasses import dataclass
+from typing import List, Optional, Tuple, Union
+
+import math
+import numpy.typing as npt
+import torch
+from torch import nn
+
+from transformers import PreTrainedModel
+from transformers import T5Config, T5Model
+from transformers.utils import logging
+
+from .configuration_moment import MomentConfig
+
+logger = logging.get_logger(__name__)
+
+@dataclass
+class TimeseriesOutputs:
+    # forecast: npt.NDArray = None
+    # anomaly_scores: npt.NDArray = None
+    logits: npt.NDArray = None
+    labels: int = None
+    input_mask: npt.NDArray = None
+    pretrain_mask: npt.NDArray = None
+    # reconstruction: npt.NDArray = None
+    embeddings: npt.NDArray = None
+    metadata: dict = None
+    illegal_output: bool = False
+    hidden_states: npt.NDArray = None  # For Mists model
+    input_mask_patch_view: npt.NDArray = None  # For Mists model
+
+
+# refers: https://github.com/moment-timeseries-foundation-model/moment/blob/088b253a1138ac7e48a7efc9bf902336c9eec8d9/momentfm/utils/masking.py#L6C1-L6C2
+class Masking:
+    def __init__(
+        self, mask_ratio: float = 0.3, patch_len: int = 8, stride: Optional[int] = None
+    ):
+        """
+        Indices with 0 mask are hidden, and with 1 are observed.
+        """
+        self.mask_ratio = mask_ratio
+        self.patch_len = patch_len
+        self.stride = patch_len if stride is None else stride
+
+    @staticmethod
+    def convert_seq_to_patch_view(
+        mask: torch.Tensor, patch_len: int = 8, stride: Optional[int] = None
+    ):
+        """
+        Input:
+            mask : torch.Tensor of shape [batch_size x seq_len]
+        Output
+            mask : torch.Tensor of shape [batch_size x n_patches]
+        """
+        stride = patch_len if stride is None else stride
+        mask = mask.unfold(dimension=-1, size=patch_len, step=stride)
+        # mask : [batch_size x n_patches x patch_len]
+        return (mask.sum(dim=-1) == patch_len).long()
+
+    @staticmethod
+    def convert_patch_to_seq_view(
+        mask: torch.Tensor,
+        patch_len: int = 8,
+    ):
+        """
+        Input:
+            mask : torch.Tensor of shape [batch_size x n_patches]
+        Output:
+            mask : torch.Tensor of shape [batch_size x seq_len]
+        """
+        return mask.repeat_interleave(patch_len, dim=-1)
+
+    def generate_mask(self, x: torch.Tensor, input_mask: Optional[torch.Tensor] = None):
+        """
+        Input:
+            x : torch.Tensor of shape
+            [batch_size x n_channels x n_patches x patch_len] or
+            [batch_size x n_channels x seq_len]
+            input_mask: torch.Tensor of shape [batch_size x seq_len] or
+            [batch_size x n_patches]
+        Output:
+            mask : torch.Tensor of shape [batch_size x seq_len]
+        """
+        if x.ndim == 4:
+            return self._mask_patch_view(x, input_mask=input_mask)
+        elif x.ndim == 3:
+            return self._mask_seq_view(x, input_mask=input_mask)
+
+    def _mask_patch_view(self, x, input_mask=None):
+        """
+        Input:
+            x : torch.Tensor of shape
+            [batch_size x n_channels x n_patches x patch_len]
+            input_mask: torch.Tensor of shape [batch_size x seq_len]
+        Output:
+            mask : torch.Tensor of shape [batch_size x n_patches]
+        """
+        input_mask = self.convert_seq_to_patch_view(
+            input_mask, self.patch_len, self.stride
+        )
+        n_observed_patches = input_mask.sum(dim=-1, keepdim=True)  # batch_size x 1
+
+        batch_size, _, n_patches, _ = x.shape
+        len_keep = torch.ceil(n_observed_patches * (1 - self.mask_ratio)).long()
+        noise = torch.rand(
+            batch_size, n_patches, device=x.device
+        )  # noise in [0, 1], batch_size x n_channels x n_patches
+        noise = torch.where(
+            input_mask == 1, noise, torch.ones_like(noise)
+        )  # only keep the noise of observed patches
+
+        # Sort noise for each sample
+        ids_shuffle = torch.argsort(
+            noise, dim=1
+        )  # Ascend: small is keep, large is remove
+        ids_restore = torch.argsort(
+            ids_shuffle, dim=1
+        )  # ids_restore: [batch_size x n_patches]
+
+        # Generate the binary mask: 0 is keep, 1 is remove
+        mask = torch.zeros(
+            [batch_size, n_patches], device=x.device
+        )  # mask: [batch_size x n_patches]
+        for i in range(batch_size):
+            mask[i, : len_keep[i]] = 1
+
+        # Unshuffle to get the binary mask
+        mask = torch.gather(mask, dim=1, index=ids_restore)
+
+        return mask.long()
+
+    def _mask_seq_view(self, x, input_mask=None):
+        """
+        Input:
+            x : torch.Tensor of shape
+            [batch_size x n_channels x seq_len]
+            input_mask: torch.Tensor of shape [batch_size x seq_len]
+        Output:
+            mask : torch.Tensor of shape [batch_size x seq_len]
+        """
+        x = x.unfold(dimension=-1, size=self.patch_len, step=self.stride)
+        mask = self._mask_patch_view(x, input_mask=input_mask)
+        return self.convert_patch_to_seq_view(mask, self.patch_len).long()
+
+
+# refers: https://github.com/moment-timeseries-foundation-model/moment/blob/088b253a1138ac7e48a7efc9bf902336c9eec8d9/momentfm/models/layers/revin.py#L5
+def nanvar(tensor, dim=None, keepdim=False):
+    tensor_mean = tensor.nanmean(dim=dim, keepdim=True)
+    output = (tensor - tensor_mean).square().nanmean(dim=dim, keepdim=keepdim)
+    return output
+
+# refers: https://github.com/moment-timeseries-foundation-model/moment/blob/088b253a1138ac7e48a7efc9bf902336c9eec8d9/momentfm/models/layers/revin.py#L11
+def nanstd(tensor, dim=None, keepdim=False):
+    output = nanvar(tensor, dim=dim, keepdim=keepdim)
+    output = output.sqrt()
+    return output
+
+# refers: https://github.com/moment-timeseries-foundation-model/moment/blob/088b253a1138ac7e48a7efc9bf902336c9eec8d9/momentfm/models/layers/revin.py#L17
+class RevIN(nn.Module):
+    def __init__(self, num_features: int, eps: float = 1e-5, affine: bool = False):
+        """
+        :param num_features: the number of features or channels
+        :param eps: a value added for numerical stability
+        :param affine: if True, RevIN has learnable affine parameters
+        """
+        super(RevIN, self).__init__()
+        self.num_features = num_features
+        self.eps = eps
+        self.affine = affine
+
+        if self.affine:
+            self._init_params()
+
+    def forward(self, x: torch.Tensor, mode: str = "norm", mask: torch.Tensor = None):
+        """
+        :param x: input tensor of shape (batch_size, n_channels, seq_len)
+        :param mode: 'norm' or 'denorm'
+        :param mask: input mask of shape (batch_size, seq_len)
+        :return: RevIN transformed tensor
+        """
+        if mode == "norm":
+            self._get_statistics(x, mask=mask)
+            x = self._normalize(x)
+        elif mode == "denorm":
+            x = self._denormalize(x)
+        else:
+            raise NotImplementedError
+        return x
+
+    def _init_params(self):
+        # initialize RevIN params: (C,)
+        self.affine_weight = nn.Parameter(torch.ones(1, self.num_features, 1))
+        self.affine_bias = nn.Parameter(torch.zeros(1, self.num_features, 1))
+
+    def _get_statistics(self, x, mask=None):
+        """
+        x    : batch_size x n_channels x seq_len
+        mask : batch_size x seq_len
+        """
+        if mask is None:
+            mask = torch.ones((x.shape[0], x.shape[-1]))
+        n_channels = x.shape[1]
+        mask = mask.unsqueeze(1).repeat(1, n_channels, 1).bool()
+        # Set masked positions to NaN, and unmasked positions are taken from x
+        masked_x = torch.where(mask, x, torch.nan)
+        self.mean = torch.nanmean(masked_x, dim=-1, keepdim=True).detach()
+        self.stdev = nanstd(masked_x, dim=-1, keepdim=True).detach() + self.eps
+        # self.stdev = torch.sqrt(
+        #     torch.var(masked_x, dim=-1, keepdim=True) + self.eps).get_data().detach()
+        # NOTE: By default not bessel correction
+
+    def _normalize(self, x):
+        x = x - self.mean
+        x = x / self.stdev
+
+        if self.affine:
+            x = x * self.affine_weight
+            x = x + self.affine_bias
+        return x
+
+    def _denormalize(self, x):
+        if self.affine:
+            x = x - self.affine_bias
+            x = x / (self.affine_weight + self.eps * self.eps)
+        x = x * self.stdev
+        x = x + self.mean
+        return x
+
+
+# refers: https://github.com/moment-timeseries-foundation-model/moment/blob/088b253a1138ac7e48a7efc9bf902336c9eec8d9/momentfm/models/layers/embed.py#L10
+class PositionalEmbedding(nn.Module):
+    def __init__(self, d_model, max_len=5000, model_name="MOMENT"):
+        super(PositionalEmbedding, self).__init__()
+        self.model_name = model_name
+
+        # Compute the positional encodings once in log space.
+        pe = torch.zeros(max_len, d_model).float()
+        pe.require_grad = False
+
+        position = torch.arange(0, max_len).float().unsqueeze(1)
+        div_term = (
+            torch.arange(0, d_model, 2).float() * -(math.log(10000.0) / d_model)
+        ).exp()
+
+        pe[:, 0::2] = torch.sin(position * div_term)
+        pe[:, 1::2] = torch.cos(position * div_term)
+
+        pe = pe.unsqueeze(0)
+        self.register_buffer("pe", pe)
+
+    def forward(self, x):
+        if (
+            self.model_name == "MOMENT"
+            or self.model_name == "TimesNet"
+            or self.model_name == "GPT4TS"
+        ):
+            return self.pe[:, : x.size(2)]
+        else:
+            return self.pe[:, : x.size(1)]
+        
+
+# refers: https://github.com/moment-timeseries-foundation-model/moment/blob/088b253a1138ac7e48a7efc9bf902336c9eec8d9/momentfm/models/layers/embed.py#L181
+class PatchEmbedding(nn.Module):
+    def __init__(
+        self,
+        d_model: int = 768,
+        seq_len: int = 512,
+        patch_len: int = 8,
+        stride: int = 8,
+        dropout: int = 0.1,
+        add_positional_embedding: bool = False,
+        value_embedding_bias: bool = False,
+        orth_gain: float = 1.41,
+    ):
+        super(PatchEmbedding, self).__init__()
+        self.patch_len = patch_len
+        self.seq_len = seq_len
+        self.stride = stride
+        self.d_model = d_model
+        self.add_positional_embedding = add_positional_embedding
+
+        self.value_embedding = nn.Linear(patch_len, d_model, bias=value_embedding_bias)
+        self.mask_embedding = nn.Parameter(torch.zeros(d_model))
+
+        if orth_gain is not None:
+            torch.nn.init.orthogonal_(self.value_embedding.weight, gain=orth_gain)
+            if value_embedding_bias:
+                self.value_embedding.bias.data.zero_()
+            # torch.nn.init.orthogonal_(self.mask_embedding, gain=orth_gain) # Fails
+
+        # Positional embedding
+        if self.add_positional_embedding:
+            self.position_embedding = PositionalEmbedding(d_model)
+
+        # Residual dropout
+        self.dropout = nn.Dropout(dropout)
+
+    def forward(self, x: torch.Tensor, mask: torch.Tensor = None) -> torch.Tensor:
+        mask = Masking.convert_seq_to_patch_view(
+            mask, patch_len=self.patch_len
+        ).unsqueeze(-1)
+        # mask : [batch_size x n_patches x 1]
+        n_channels = x.shape[1]
+        mask = (
+            mask.repeat_interleave(self.d_model, dim=-1)
+            .unsqueeze(1)
+            .repeat(1, n_channels, 1, 1)
+        )
+        # mask : [batch_size x n_channels x n_patches x d_model]
+
+        # Input encoding
+        x = mask * self.value_embedding(x) + (1 - mask) * self.mask_embedding
+        if self.add_positional_embedding:
+            x = x + self.position_embedding(x)
+
+        return self.dropout(x)
+
+
+# refers: https://github.com/moment-timeseries-foundation-model/moment/blob/088b253a1138ac7e48a7efc9bf902336c9eec8d9/momentfm/models/layers/embed.py#L237C1-L251C17
+class Patching(nn.Module):
+    def __init__(self, patch_len: int, stride: int):
+        super().__init__()
+        self.patch_len = patch_len
+        self.stride = stride
+        if self.stride != self.patch_len:
+            logger.warning(
+                "Stride and patch length are not equal. "
+                "This may lead to unexpected behavior."
+            )
+
+    def forward(self, x):
+        x = x.unfold(dimension=-1, size=self.patch_len, step=self.stride)
+        # x : [batch_size x n_channels x num_patch x patch_len]
+        return x
+
+
+class MomentPreTrainedModel(PreTrainedModel):
+    config_class = MomentConfig
+
+    base_model_prefix = "model"
+    supports_gradient_checkpointing = True
+    _no_split_modules = ["T5Block"]
+    _skip_keys_device_placement = ""
+    
+    # 本来のT5の_init_weightsはもっと詳細だが、事前学習の予定はないためここでは簡単にしている。
+    # refers: https://github.com/huggingface/transformers/blob/517df566f572d90e6301df87870f651f0d1b1110/src/transformers/models/t5/modeling_t5.py#L810
+    def _init_weights(self, module):
+        std = self.config.t5_config["initializer_factor"]
+        if isinstance(module, nn.Linear):
+            module.weight.data.normal_(mean=0.0, std=std)
+            if module.bias is not None:
+                module.bias.data.zero_()
+        elif isinstance(module, nn.Embedding):
+            module.weight.data.normal_(mean=0.0, std=std)
+            if module.padding_idx is not None:
+                module.weight.data[module.padding_idx].zero_()
+
+
+class MomentEmbeddingModel(MomentPreTrainedModel):
+    def __init__(self, config):
+        super().__init__(config)
+        self.config = config
+        self.seq_len = config.seq_len
+        self.patch_len = config.patch_len
+        self.patch_stride_len = config.patch_stride_len
+
+        # TODO: normalizer, tokenizerはProcessor側に配置するべきか？
+        # 現状の考え: 特にMomentから切り離す用途もない。
+        # 　　　　　　Processor側では入力の512timestepsへの切り取り等、
+        # 　　　　　　input validationとTensorへの切り替えを行うで良さそう。
+        self.normalizer = RevIN(
+            num_features=getattr(config, "revin_num_features", 1), eps=getattr(config, "revin_eps", 1e-5), affine=getattr(config, "revin_affine", False)
+        )
+        self.tokenizer = Patching(
+            patch_len=config.patch_len, stride=config.patch_stride_len
+        )
+        # モデル構成
+        self.patch_embedding = PatchEmbedding(
+            d_model=config.d_model,
+            seq_len=config.seq_len,
+            patch_len=config.patch_len,
+            stride=config.patch_stride_len,
+            dropout=getattr(config, "dropout", 0.1),
+            add_positional_embedding=getattr(config, "add_positional_embedding", True),
+            value_embedding_bias=getattr(config, "value_embedding_bias", False),
+            orth_gain=getattr(config, "orth_gain", 1.41),
+        )
+        self.mask_generator = Masking(mask_ratio=getattr(config, "mask_ratio", 0.0))
+        self.encoder = self._get_t5_encoder(config.t5_config, config.enable_gradient_checkpointing)
+        self.head = nn.Identity()
+
+        # Frozen parameters
+        self.freeze_embedder = getattr(config, "freeze_embedder", True)
+        self.freeze_encoder = getattr(config, "freeze_encoder", True)
+        self.freeze_head = getattr(config, "freeze_head", False)
+
+        if self.freeze_embedder:
+            self.patch_embedding = freeze_parameters(self.patch_embedding)
+        if self.freeze_encoder:
+            self.encoder = freeze_parameters(self.encoder)
+        if self.freeze_head:
+            self.head = freeze_parameters(self.head)
+
+    def _get_t5_encoder(self, config: dict, enable_gradient_checkpointing: bool) -> nn.Module:
+        # random initialize
+        # Momentでは(言語で)事前学習済みのモデルを取得することもできるようになっている
+        # refers: https://github.com/moment-timeseries-foundation-model/moment/blob/088b253a1138ac7e48a7efc9bf902336c9eec8d9/momentfm/models/moment.py#L205
+        t5_config = T5Config.from_dict(config)
+        t5_model = T5Model(t5_config)
+        t5_model_encoder = t5_model.get_encoder()
+
+        if enable_gradient_checkpointing:
+            t5_model_encoder.gradient_checkpointing_enable()
+            logger.info("Enabling gradient checkpointing.")
+
+        return t5_model_encoder
+
+    def embed(
+        self,
+        x_enc: torch.Tensor,
+        input_mask: torch.Tensor = None,
+        reduction: str = "mean",
+        **kwargs,
+    ) -> TimeseriesOutputs:
+        batch_size, n_channels, seq_len = x_enc.shape
+
+        if input_mask is None:
+            input_mask = torch.ones((batch_size, seq_len)).to(x_enc.device)
+
+        x_enc = self.normalizer(x=x_enc, mask=input_mask, mode="norm")
+        x_enc = torch.nan_to_num(x_enc, nan=0, posinf=0, neginf=0)
+
+        # [batch_size x n_patches]
+        input_mask_patch_view = Masking.convert_seq_to_patch_view(
+            input_mask, self.patch_len
+        )
+
+        x_enc = self.tokenizer(x=x_enc)
+        enc_in = self.patch_embedding(x_enc, mask=input_mask)
+
+        n_patches = enc_in.shape[2]
+        enc_in = enc_in.reshape(
+            (batch_size * n_channels, n_patches, self.config.d_model)
+        )
+
+        patch_view_mask = Masking.convert_seq_to_patch_view(input_mask, self.patch_len)
+        attention_mask = patch_view_mask.repeat_interleave(n_channels, dim=0)
+        outputs = self.encoder(inputs_embeds=enc_in, attention_mask=attention_mask)
+        enc_out = outputs.last_hidden_state
+        hidden_states = outputs.last_hidden_state  # hidden_statesを取得
+
+        enc_out = enc_out.reshape((-1, n_channels, n_patches, self.config.d_model))
+        # [batch_size x n_channels x n_patches x d_model]
+
+        if reduction == "mean":
+            enc_out = enc_out.mean(dim=1, keepdim=False)  # Mean across channels
+            # [batch_size x n_patches x d_model]
+            input_mask_patch_view = input_mask_patch_view.unsqueeze(-1).repeat(
+                1, 1, self.config.d_model
+            )
+            enc_out = (input_mask_patch_view * enc_out).sum(
+                dim=1
+            ) / input_mask_patch_view.sum(dim=1)
+        else:
+            raise NotImplementedError(f"Reduction method {reduction} not implemented.")
+        
+        # For Mists model
+        # [batch_size, n_channels x n_patches, d_model]
+        # Ensure hidden_states are consistent for both short and long inputs with input_mask specified
+        # hidden_states = hidden_states.reshape(batch_size, n_channels, n_patches, self.config.d_model).transpose(1, 2).reshape(batch_size, -1, self.config.d_model)
+        # [batch_size x n_patches]
+        input_mask_patch_view_for_hidden_states = Masking.convert_seq_to_patch_view(input_mask, self.patch_len)
+        # [batch_size　x n_channels x n_patches x d_model]
+        input_mask_patch_view_for_hidden_states = input_mask_patch_view_for_hidden_states.unsqueeze(1).unsqueeze(-1).repeat(
+            1, n_channels, 1, self.config.d_model
+        )
+        # [batch_size x n_channels x n_patches x d_model]
+        hidden_states = hidden_states.reshape(batch_size, n_channels, n_patches, self.config.d_model)
+        hidden_states = input_mask_patch_view_for_hidden_states * hidden_states
+        # [batch_size, n_channels x n_patches, d_model]
+        hidden_states = hidden_states.transpose(1, 2).reshape(batch_size, -1, self.config.d_model)
+        
+        # [batch_size x n_patches]
+        input_mask_patch_view_for_mists = Masking.convert_seq_to_patch_view(input_mask, self.patch_len)
+        # [batch_size, n_channels x n_patches]
+        input_mask_patch_view_for_mists = input_mask_patch_view_for_mists.repeat_interleave(n_channels, dim=1)
+
+        return TimeseriesOutputs(
+            embeddings=enc_out, input_mask=input_mask, metadata=reduction, hidden_states=hidden_states, input_mask_patch_view=input_mask_patch_view_for_mists
+        )
+    
+    def forward(
+        self,
+        time_series_values: torch.Tensor,
+        # mask: torch.Tensor = None,
+        input_mask: torch.Tensor = None,
+        **kwargs,
+    ) -> TimeseriesOutputs:
+        if input_mask is None:
+            input_mask = torch.ones_like(time_series_values[:, 0, :])
+
+        return self.embed(x_enc=time_series_values, input_mask=input_mask, **kwargs)
+    
+    def calculate_n_patches(self, seq_len: int) -> int:
+        """
+        時系列の長さ（seq_len）を与えて、モデルのself.patch_lenとself.strideを使ってn_patchesを計算して返します。
+        strideがNoneの場合はpatch_lenを使用します。
+
+        Args:
+            seq_len (int): 時系列の長さ
+
+        Returns:
+            int: 計算されたn_patchesの数
+        """
+        stride = self.patch_stride_len if self.patch_stride_len is not None else self.patch_len
+        n_patches = (seq_len - self.patch_len) // stride + 1
+        return n_patches
+
+
+# refers: https://github.com/moment-timeseries-foundation-model/moment/blob/088b253a1138ac7e48a7efc9bf902336c9eec8d9/momentfm/models/moment.py#L601
+def freeze_parameters(model):
+    """
+    Freeze parameters of the model
+    """
+    # Freeze the parameters
+    for name, param in model.named_parameters():
+        param.requires_grad = False
+
+    return model