End of training

20250510_111509.log

@@ -0,0 +1,27 @@
+[2025-05-10 11:15:11] Created output directory: train_results_pred_mask/google_gemma-3-1b-pt_ds1000_upsample1000_predict_mask
+[2025-05-10 11:15:11] Chat mode disabled
+[2025-05-10 11:15:11] Model size is 3B or smaller (1 B). Using full fine-tuning.
+[2025-05-10 11:15:11] No QA format data will be used
+[2025-05-10 11:15:11] Limiting dataset size to: 1000 samples
+[2025-05-10 11:15:12] =======================================
+[2025-05-10 11:15:12] Starting training for model: google/gemma-3-1b-pt
+[2025-05-10 11:15:12] =======================================
+[2025-05-10 11:15:12] CUDA_VISIBLE_DEVICES: 0,1,2,3
+[2025-05-10 11:15:12] WANDB_PROJECT: wikidyk-ar
+[2025-05-10 11:15:12] DATA_PATH: data/wikidyk2022-2025_01082025_gpt-4o_evalv2_pages_formatted_combined_v2.json
+[2025-05-10 11:15:12] Global Batch Size: 128
+[2025-05-10 11:15:12] Data Size: 1000
+[2025-05-10 11:15:12] Executing command: torchrun --nproc_per_node "4" --master-port 29501 src/train.py       --model_name_or_path "google/gemma-3-1b-pt"       --data_path "data/wikidyk2022-2025_01082025_gpt-4o_evalv2_pages_formatted_combined_v2.json"       --output_dir "train_results_pred_mask/google_gemma-3-1b-pt_ds1000_upsample1000_predict_mask"       --num_upsample "1000"       --per_device_train_batch_size "32"       --gradient_accumulation_steps "1"       --learning_rate "2e-5"       --num_train_epochs "1"       --model_max_length "32768"       --report_to wandb --logging_steps 50 --save_strategy no       --bf16 True --use_flash_attention_2 True       --qa_data_ratio "-1"       --predict_mask "true"                            --ds_size 1000
+[2025-05-10 11:15:12] Training started at Sat May 10 11:15:12 UTC 2025
+[2025-05-10 11:15:13] ERROR: Training failed for google/gemma-3-1b-pt with exit code 1
+[2025-05-10 11:15:13] ERROR: Training failed for google/gemma-3-1b-pt with exit code 1
+[2025-05-10 11:15:13] Check error log for details: train_results_pred_mask/google_gemma-3-1b-pt_ds1000_upsample1000_predict_mask/20250510_111509.log
+[2025-05-10 11:15:13] Resource usage after training google/gemma-3-1b-pt:
+[2025-05-10 11:15:13] GPU memory usage:
+3635 MiB, 40960 MiB
+3615 MiB, 40960 MiB
+3619 MiB, 40960 MiB
+3611 MiB, 40960 MiB
+[2025-05-10 11:15:13] Disk space usage for model outputs:
+4.0K	train_results_pred_mask/google_gemma-3-1b-pt_ds1000_upsample1000_predict_mask
+[2025-05-10 11:15:13] 

README.md

@@ -0,0 +1,57 @@
+---
+library_name: transformers
+license: gemma
+base_model: google/gemma-3-1b-pt
+tags:
+- generated_from_trainer
+model-index:
+- name: google_gemma-3-1b-pt_ds1000_upsample1000_predict_mask
+  results: []
+---
+
+<!-- This model card has been generated automatically according to the information the Trainer had access to. You
+should probably proofread and complete it, then remove this comment. -->
+
+# google_gemma-3-1b-pt_ds1000_upsample1000_predict_mask
+
+This model is a fine-tuned version of [google/gemma-3-1b-pt](https://huggingface.co/google/gemma-3-1b-pt) on an unknown dataset.
+
+## Model description
+
+More information needed
+
+## Intended uses & limitations
+
+More information needed
+
+## Training and evaluation data
+
+More information needed
+
+## Training procedure
+
+### Training hyperparameters
+
+The following hyperparameters were used during training:
+- learning_rate: 2e-05
+- train_batch_size: 32
+- eval_batch_size: 8
+- seed: 42
+- distributed_type: multi-GPU
+- num_devices: 4
+- total_train_batch_size: 128
+- total_eval_batch_size: 32
+- optimizer: Use adamw_torch with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
+- lr_scheduler_type: linear
+- num_epochs: 1.0
+
+### Training results
+
+
+
+### Framework versions
+
+- Transformers 4.51.3
+- Pytorch 2.6.0+cu124
+- Datasets 3.5.1
+- Tokenizers 0.21.1

added_tokens.json

@@ -0,0 +1,3 @@
+{
+  "<image_soft_token>": 262144
+}

config.json

@@ -0,0 +1,34 @@
+{
+  "architectures": [
+    "Gemma3ForCausalLM"
+  ],
+  "attention_bias": false,
+  "attention_dropout": 0.0,
+  "attn_logit_softcapping": null,
+  "bos_token_id": 2,
+  "cache_implementation": "hybrid",
+  "eos_token_id": 1,
+  "final_logit_softcapping": null,
+  "head_dim": 256,
+  "hidden_activation": "gelu_pytorch_tanh",
+  "hidden_size": 1152,
+  "initializer_range": 0.02,
+  "intermediate_size": 6912,
+  "max_position_embeddings": 32768,
+  "model_type": "gemma3_text",
+  "num_attention_heads": 4,
+  "num_hidden_layers": 26,
+  "num_key_value_heads": 1,
+  "pad_token_id": 0,
+  "query_pre_attn_scalar": 256,
+  "rms_norm_eps": 1e-06,
+  "rope_local_base_freq": 10000,
+  "rope_scaling": null,
+  "rope_theta": 1000000,
+  "sliding_window": 512,
+  "sliding_window_pattern": 6,
+  "torch_dtype": "bfloat16",
+  "transformers_version": "4.51.3",
+  "use_cache": true,
+  "vocab_size": 262144
+}

generation_config.json

@@ -0,0 +1,13 @@
+{
+  "bos_token_id": 2,
+  "cache_implementation": "hybrid",
+  "do_sample": true,
+  "eos_token_id": [
+    1,
+    106
+  ],
+  "pad_token_id": 0,
+  "top_k": 64,
+  "top_p": 0.95,
+  "transformers_version": "4.51.3"
+}

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:193463484d2022a9d7dba34a9028853b637994417d9b17a44949d28dcabe431c
+size 1999811208

special_tokens_map.json

@@ -0,0 +1,33 @@
+{
+  "boi_token": "<start_of_image>",
+  "bos_token": {
+    "content": "<bos>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "eoi_token": "<end_of_image>",
+  "eos_token": {
+    "content": "<eos>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "image_token": "<image_soft_token>",
+  "pad_token": {
+    "content": "<pad>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "unk_token": {
+    "content": "<unk>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  }
+}

tokenizer.model

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:1299c11d7cf632ef3b4e11937501358ada021bbdf7c47638d13c0ee982f2e79c
+size 4689074

training_args.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:5d7a0ce79bb181a18ce66142bd7b3fcd8acddc8d0a118592bd81892b1dfeebd7
+size 5432