upload

.gitattributes

@@ -33,3 +33,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+*.json filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,76 @@
+---
+license: llama2
+base_model: meta-llama/CodeLlama-34b-Instruct-hf
+tags:
+- alignment-handbook
+- generated_from_trainer
+datasets:
+- meng-lab/CodeLlama-34B-Instruct-gsm8k
+model-index:
+- name: CodeLlama-34b-Instruct-sft-5e-3-epoch-100-gsm8k
+  results: []
+---
+
+<!-- This model card has been generated automatically according to the information the Trainer had access to. You
+should probably proofread and complete it, then remove this comment. -->
+
+[<img src="https://raw.githubusercontent.com/wandb/assets/main/wandb-github-badge-28.svg" alt="Visualize in Weights & Biases" width="200" height="32"/>](https://wandb.ai/uva-llm/huggingface/runs/nemwhwry)
+# CodeLlama-34b-Instruct-sft-5e-3-epoch-100-gsm8k
+
+This model is a fine-tuned version of [meta-llama/CodeLlama-34b-Instruct-hf](https://huggingface.co/meta-llama/CodeLlama-34b-Instruct-hf) on the meng-lab/CodeLlama-34B-Instruct-gsm8k dataset.
+It achieves the following results on the evaluation set:
+- Loss: 4.0230
+- Loss Layer 6 Head: 1.2898
+- Loss Layer 12 Head: 1.0049
+- Loss Layer 18 Head: 0.9093
+- Loss Layer 24 Head: 0.4408
+- Loss Layer 30 Head: 0.2683
+- Loss Layer 36 Head: 0.1391
+- Loss Layer 42 Head: 0.0639
+
+## Model description
+
+More information needed
+
+## Intended uses & limitations
+
+More information needed
+
+## Training and evaluation data
+
+More information needed
+
+## Training procedure
+
+### Training hyperparameters
+
+The following hyperparameters were used during training:
+- learning_rate: 0.005
+- train_batch_size: 1
+- eval_batch_size: 2
+- seed: 42
+- distributed_type: multi-GPU
+- num_devices: 8
+- gradient_accumulation_steps: 16
+- total_train_batch_size: 128
+- total_eval_batch_size: 16
+- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
+- lr_scheduler_type: cosine
+- lr_scheduler_warmup_ratio: 0.1
+- num_epochs: 100
+
+### Training results
+
+| Training Loss | Epoch   | Step | Validation Loss | Loss Layer 6 Head | Loss Layer 12 Head | Loss Layer 18 Head | Loss Layer 24 Head | Loss Layer 30 Head | Loss Layer 36 Head | Loss Layer 42 Head |
+|:-------------:|:-------:|:----:|:---------------:|:-----------------:|:------------------:|:------------------:|:------------------:|:------------------:|:------------------:|:------------------:|
+| 2.6241        | 25.8065 | 200  | 4.3768          | 1.3707            | 1.0927             | 0.9492             | 0.4907             | 0.2888             | 0.1534             | 0.0899             |
+| 1.6189        | 51.6129 | 400  | 4.0476          | 1.3067            | 0.9916             | 0.9104             | 0.4445             | 0.2716             | 0.1405             | 0.0663             |
+| 1.3737        | 77.4194 | 600  | 4.0230          | 1.2898            | 1.0049             | 0.9093             | 0.4408             | 0.2683             | 0.1391             | 0.0639             |
+
+
+### Framework versions
+
+- Transformers 4.43.2
+- Pytorch 2.1.2
+- Datasets 3.2.0
+- Tokenizers 0.19.1

added_tokens.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:9fda451030df30d6b04d32f8028647d1f321af096ab028cae5626f98591e3fbf
+size 87

all_results.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:997aa1abed4aa47fccaf4da2b675d9e8cda0d05beb949ca376102b106f16e4a5
+size 248

config.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:7ee43757293d227dced977b2e44aa164695055a87d9cea574b6b1c47ffa11bcc
+size 759

generation_config.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:b8efb27b6a9f4c441cb9a76e40fecd2b6273a4bb39b9b23bfde3cb65d64a9297
+size 165

model-00001-of-00015.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:679949bfffd2f9623f633940999081f545a0edde4e72cd0311921a13dea48b08
+size 4978740880

model-00002-of-00015.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:ed3e10b53fc5a5720dcc695e88701bb36d2d71762bc2363e8a18c0f1089325e5
+size 4873882928

model-00003-of-00015.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a1689fe8677db9a3432cd5cf89b2bf89ad2479f915604b552352d29c475c1b80
+size 4815196024

model-00004-of-00015.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:ee8b309dbd65e7d6e87e46c1ecb2e39b102254c9304ce7b31f85422e57ada8ba
+size 4873882952

model-00005-of-00015.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:aa7f94835f32247b9a492ba1d02d67ca6cca05a5fb2cedf3b9ce1900366505f9
+size 4815196056

model-00006-of-00015.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:6f6169284f5fe81ab23accfa0125153c25a94cded2f660030118f58aebc75162
+size 4873882952

model-00007-of-00015.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:2ad40e10bfa40896b79a7ff3e3a6380f8a6bde26a7dd020b162fed68e195ced9
+size 4815196056

model-00008-of-00015.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:dc715d3442c978e90a70fe4c83909f473c6d2ee150257daf689fcf791a06c3f4
+size 4873882952

model-00009-of-00015.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:054b1b0621bb7a14b2bd53c1599878340b8642c5e81122bbcf59ecbaed5f1959
+size 4815196056

model-00010-of-00015.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:54414757f8da5530b13b420fac33e8fcff6324aa05d6f21db269da7ec1da95ac
+size 4873882952

model-00011-of-00015.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:88f48912b3a7557533dd8b9a875f1f5768f6085d19e6f261b51e8713cd08b600
+size 4815196056

model-00012-of-00015.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:c7992ffad70d218e62473e74917c04e4f7703864edb9ece8bbe0832b98539f41
+size 4873882952

model-00013-of-00015.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:cd1d3d7b3a69e0024406617ca43f291af99a517cd3746bd10806dcb2e851657b
+size 4815196056

model-00014-of-00015.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:999fe25983a33afe845938cbf70f58e3b4182fca6027656b1be9d6c681123349
+size 4911648032

model-00015-of-00015.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:c7c4845c17c6d60ee34866795187bef798d391b8ba9ab98d4e7587058b7de00b
+size 402653568

model.safetensors.index.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:fefe63f5750cd25d75368e6736230d3c2e573c9c9f65ecf8944c72211d6ae9ef
+size 36411

special_tokens_map.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:8f32ca2dd65336aceb9e359886f34f214330137c9e801e8f41cf542287fc2630
+size 538

tokenizer.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:2f83209f47d970327175fab9bfb5a05d74da1fff4ada58967223d643e4356e5c
+size 1843531

tokenizer.model

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:9e556afd44213b6bd1be2b850ebbbd98f5481437a8021afaf58ee7fb1818d347
+size 499723

tokenizer_config.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:3ad15334ddd934f433f500267010a3ad3f14497a4a2cde66b6c77a8a5c60ff6f
+size 2652

train_results.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:997aa1abed4aa47fccaf4da2b675d9e8cda0d05beb949ca376102b106f16e4a5
+size 248

trainer_state.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:5071893907d965261542a164452287bd0caea3bddabcab55c727dfa369dff1e8
+size 74638

training_args.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a9fad367b497efcfeb633a1276c7e3f8bf57b4281f898be44851f9dedd517802
+size 6904